[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:14 dist-test-slave-ncp9 krb5kdc[29058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:14 dist-test-slave-ncp9 krb5kdc[29058](info): set up 2 sockets
Aug 06 04:29:14 dist-test-slave-ncp9 krb5kdc[29058](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 04:29:15.808516 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.009s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:15 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:15.942415 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:35101
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:16.083177 29077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:16.083523 29077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:16.083568 29077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:16.088030 29077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:16.088126 29077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:16.088145 29077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:16.088161 29077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:16.088176 29077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:16.094574 29077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:16.096292 29077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:16.097864 29077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:16.112416 29087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:16.112573 29077 server_base.cc:1034] running on GCE node
W20260806 04:29:16.112614 29086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:16.112421 29089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:16.113771 29077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:16.115852 29077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:16.117112 29077 hybrid_clock.cc:648] HybridClock initialized: now 1785990556117079 us; error 68 us; skew 500 ppm
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990556, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:16.120448 29077 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:16.121846 29077 webserver.cc:460] Webserver started at http://127.28.94.62:34427/ using document root <none> and password file <none>
I20260806 04:29:16.122959 29077 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:16.123070 29077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:16.123543 29077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:16.126855 29077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5abcbdf44bd2416781d4beff3d2c515a"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "eafdf3d8afa774ab8fc991b9299e3491"
server_key_iv: "137479e73ccce7d615a2449d0cdd82a7"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.127519 29077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5abcbdf44bd2416781d4beff3d2c515a"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "eafdf3d8afa774ab8fc991b9299e3491"
server_key_iv: "137479e73ccce7d615a2449d0cdd82a7"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.133736 29077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:16.138190 29100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.140184 29077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:16.140426 29077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5abcbdf44bd2416781d4beff3d2c515a"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "eafdf3d8afa774ab8fc991b9299e3491"
server_key_iv: "137479e73ccce7d615a2449d0cdd82a7"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.140585 29077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:16.173599 29077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:16.177899 29077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:16.178124 29077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:16.195013 29077 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:35101
I20260806 04:29:16.195119 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:35101 every 8 connection(s)
I20260806 04:29:16.196704 29077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:16.202486 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29077
I20260806 04:29:16.202584 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:16.202653 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:16.204221 29048 external_mini_cluster.cc:1428] Setting key c0d7d9f2858d5e81a5e3bb9303b41ebb
I20260806 04:29:16.213474 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: Bootstrap starting.
I20260806 04:29:16.217453 29166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:16.219170 29166 log.cc:824] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:16.221903 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: No bootstrap required, opened a new log
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:16.228950 29166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } }
I20260806 04:29:16.229331 29166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:16.229445 29166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abcbdf44bd2416781d4beff3d2c515a, State: Initialized, Role: FOLLOWER
I20260806 04:29:16.230091 29166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } }
I20260806 04:29:16.230257 29166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:16.230369 29166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:16.230491 29166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:16.231802 29166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } }
I20260806 04:29:16.232342 29166 leader_election.cc:302] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5abcbdf44bd2416781d4beff3d2c515a; no voters:
I20260806 04:29:16.232744 29166 leader_election.cc:288] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:16.233055 29178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:16.233538 29178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [term 1 LEADER]: Becoming Leader. State: Replica: 5abcbdf44bd2416781d4beff3d2c515a, State: Running, Role: LEADER
I20260806 04:29:16.234028 29178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } }
I20260806 04:29:16.234817 29166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:16.237588 29180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5abcbdf44bd2416781d4beff3d2c515a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } } }
I20260806 04:29:16.237764 29180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:16.238153 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:16.209363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.28.94.62:35101)
0806 04:29:16.210029 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:16.210090 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:16.211309 (+ 1219us) server_negotiation.cc:407] Connection header received
0806 04:29:16.216411 (+ 5102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:16.216434 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:16.216962 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:16.217340 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:16.220529 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.221829 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.223325 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.223917 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.224276 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:16.229785 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:16.229829 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:16.229846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:16.229892 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:16.232204 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.233006 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.233014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.233021 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.233279 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.233795 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.233798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.233800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.234426 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:16.234660 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:16.235546 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 04:29:16.235904 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":201,"threads_started":1}
I20260806 04:29:16.238225 29182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5abcbdf44bd2416781d4beff3d2c515a. Latest consensus state: current_term: 1 leader_uuid: "5abcbdf44bd2416781d4beff3d2c515a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5abcbdf44bd2416781d4beff3d2c515a" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35101 } } }
I20260806 04:29:16.238348 29182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a [sys.catalog]: This master's current role is: LEADER
W20260806 04:29:16.244415 29194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:29:16.244563 29194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:29:16.244730 29196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:16.246100 29196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:16.254467 29196 catalog_manager.cc:1347] Generated new cluster ID: 16fc6f431f314b53903c19b5bbe3efbb
I20260806 04:29:16.254559 29196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:16.263933 29196 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:16.265444 29196 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:16.282418 29196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5abcbdf44bd2416781d4beff3d2c515a: Generated new TSK 0
I20260806 04:29:16.283352 29196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:16.325449 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:16.505899 29203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:16.506218 29203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:16.506266 29203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:16.510545 29203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:16.510617 29203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:16.510680 29203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:16.516815 29203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:16.518088 29203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:16.519148 29203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:16.531388 29210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:16.531428 29213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:16.531430 29211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:16.531829 29203 server_base.cc:1034] running on GCE node
I20260806 04:29:16.532225 29203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:16.532886 29203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:16.534080 29203 hybrid_clock.cc:648] HybridClock initialized: now 1785990556534062 us; error 33 us; skew 500 ppm
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990556, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:16.537464 29203 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:16.538465 29203 webserver.cc:460] Webserver started at http://127.28.94.1:45231/ using document root <none> and password file <none>
I20260806 04:29:16.539043 29203 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:16.539112 29203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:16.539335 29203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:16.541707 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "44a0699c73da471cade1cf4d5789955f"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "373eaa8b090b747c304c7d070fe2a476"
server_key_iv: "fcac772205d2caa28cb067a5ecf23c59"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.542300 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "44a0699c73da471cade1cf4d5789955f"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "373eaa8b090b747c304c7d070fe2a476"
server_key_iv: "fcac772205d2caa28cb067a5ecf23c59"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.546597 29203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:16.549278 29221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.550494 29203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:29:16.550608 29203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "44a0699c73da471cade1cf4d5789955f"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "373eaa8b090b747c304c7d070fe2a476"
server_key_iv: "fcac772205d2caa28cb067a5ecf23c59"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.550716 29203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:16.570381 29203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:16.573526 29203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:16.573745 29203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:16.574529 29203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:16.575755 29203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:16.575810 29203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.575868 29203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:16.575896 29203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.591115 29203 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:45227
I20260806 04:29:16.591171 29381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:45227 every 8 connection(s)
I20260806 04:29:16.592088 29203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:16.595427 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29203
I20260806 04:29:16.595526 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:16.595779 29048 external_mini_cluster.cc:1428] Setting key 1d1480a123215e561a66572d25c88e5c
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990556, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:16.607782 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:16.594204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:39815 (local address 127.28.94.62:35101)
0806 04:29:16.594341 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:16.594344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:16.595963 (+ 1619us) server_negotiation.cc:407] Connection header received
0806 04:29:16.597231 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:16.597235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:16.597286 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:16.597371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:16.599232 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.599881 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.601232 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.601576 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.601661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:16.604542 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:16.604545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:16.604547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:16.604571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:16.605868 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.606338 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.606342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.606343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.606415 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.606723 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.606726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.606728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.606904 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:16.606988 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:16.607487 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 04:29:16.607613 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:16.608744 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:16.594539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35101 (local address 127.28.94.1:39815)
0806 04:29:16.595793 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:16.595847 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:16.597010 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:16.597578 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:16.597588 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:16.598247 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:16.599091 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:16.599112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.600026 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.600030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:16.601113 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:16.601123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.601696 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.601698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:16.601897 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:16.602543 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:16.602558 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:16.604377 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:16.605975 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:16.605989 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:16.606005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:16.606246 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:16.606498 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:16.606502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:16.606504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:16.606637 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:16.607010 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:16.607016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:16.607255 (+ 239us) client_negotiation.cc:769] Sending connection context
0806 04:29:16.607420 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 04:29:16.607738 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":757,"threads_started":1}
I20260806 04:29:16.610261 29382 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:29:16.610526 29382 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:16.611045 29382 heartbeater.cc:511] Master 127.28.94.62:35101 requested a full tablet report, sending...
I20260806 04:29:16.613139 29121 ts_manager.cc:194] Registered new tserver with Master: 44a0699c73da471cade1cf4d5789955f (127.28.94.1:45227)
I20260806 04:29:16.615025 29121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:16.627666 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:16.771384 29388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:16.771714 29388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:16.771764 29388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:16.776120 29388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:16.776212 29388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:16.776286 29388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:16.782176 29388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:16.783519 29388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:16.784735 29388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:16.796392 29397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:16.796392 29400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:16.796416 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:16.796581 29388 server_base.cc:1034] running on GCE node
I20260806 04:29:16.797328 29388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:16.798087 29388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:16.799288 29388 hybrid_clock.cc:648] HybridClock initialized: now 1785990556799264 us; error 35 us; skew 500 ppm
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990556, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:16.802834 29388 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:16.803921 29388 webserver.cc:460] Webserver started at http://127.28.94.2:45335/ using document root <none> and password file <none>
I20260806 04:29:16.804553 29388 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:16.804659 29388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:16.804934 29388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:16.807217 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "b1ebbda2d0cf42538cbb9795022c3c4e"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "6b42f357ad71c7494e019f9cf8d0160a"
server_key_iv: "dd60ad100e75d4e6e01598cb1abe3cd8"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.807853 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ebbda2d0cf42538cbb9795022c3c4e"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "6b42f357ad71c7494e019f9cf8d0160a"
server_key_iv: "dd60ad100e75d4e6e01598cb1abe3cd8"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.812244 29388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:29:16.815114 29408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.816478 29388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:16.816633 29388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "b1ebbda2d0cf42538cbb9795022c3c4e"
format_stamp: "Formatted at 2026-08-06 04:29:16 on dist-test-slave-ncp9"
server_key: "6b42f357ad71c7494e019f9cf8d0160a"
server_key_iv: "dd60ad100e75d4e6e01598cb1abe3cd8"
server_key_version: "encryptionkey@0"
I20260806 04:29:16.816761 29388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:16.859864 29388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:16.863998 29388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:16.864243 29388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:16.865015 29388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:16.866043 29388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:16.866097 29388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.866158 29388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:16.866183 29388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:16.877899 29388 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43501
I20260806 04:29:16.877990 29565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43501 every 8 connection(s)
I20260806 04:29:16.879038 29388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:16.886234 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29388
I20260806 04:29:16.886358 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:16.886649 29048 external_mini_cluster.cc:1428] Setting key 4168d97d875bed63642bb5b6d2fa3c20
Aug 06 04:29:16 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990556, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:16.895077 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:16.881293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:42851 (local address 127.28.94.62:35101)
0806 04:29:16.881458 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:16.881462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:16.882428 (+ 966us) server_negotiation.cc:407] Connection header received
0806 04:29:16.884038 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:16.884042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:16.884123 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:16.884196 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:16.885651 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.886294 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.887715 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.887955 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.888087 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:16.891149 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:16.891154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:16.891157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:16.891193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:16.892600 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.893251 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.893255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.893257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.893355 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:16.893745 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:16.893749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:16.893751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:16.893966 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:16.894063 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:16.894740 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 04:29:16.894892 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:29:16.896024 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:16.881717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35101 (local address 127.28.94.2:42851)
0806 04:29:16.882256 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:16.882308 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:16.883781 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:16.884421 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:16.884430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:16.884866 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:16.885477 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:16.885492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.886476 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.886480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:16.887544 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:16.887559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:16.888154 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:16.888159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:16.888395 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:16.889073 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:16.889093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:16.890914 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:16.892744 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:16.892751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:16.892770 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:16.893128 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:16.893454 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:16.893456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:16.893458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:16.893624 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:16.894081 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:16.894090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:16.894422 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 04:29:16.894645 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 04:29:16.895043 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260806 04:29:16.897509 29570 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35101
I20260806 04:29:16.897850 29570 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:16.898591 29570 heartbeater.cc:511] Master 127.28.94.62:35101 requested a full tablet report, sending...
I20260806 04:29:16.899998 29121 ts_manager.cc:194] Registered new tserver with Master: b1ebbda2d0cf42538cbb9795022c3c4e (127.28.94.2:43501)
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:29:16.900641 29121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:42851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:16.920423 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:17.052305 29578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:17.052635 29578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:17.052683 29578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:17.057053 29578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:17.057137 29578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:17.057202 29578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:17.063133 29578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:17.064322 29578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:17.065448 29578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:17.077382 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:17.077404 29584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:17.077417 29588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:17.077828 29578 server_base.cc:1034] running on GCE node
I20260806 04:29:17.078348 29578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:17.079103 29578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:17.080273 29578 hybrid_clock.cc:648] HybridClock initialized: now 1785990557080248 us; error 45 us; skew 500 ppm
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:17.083362 29578 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:17.084425 29578 webserver.cc:460] Webserver started at http://127.28.94.3:37875/ using document root <none> and password file <none>
I20260806 04:29:17.085108 29578 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:17.085188 29578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:17.085426 29578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:17.087793 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "132e4d711c2e4cd69bbbc10d8c77bfca"
format_stamp: "Formatted at 2026-08-06 04:29:17 on dist-test-slave-ncp9"
server_key: "386e9f502aeede05f091befa8d232d93"
server_key_iv: "d7f452816f64feebe2b25811dfe7e42d"
server_key_version: "encryptionkey@0"
I20260806 04:29:17.088434 29578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "132e4d711c2e4cd69bbbc10d8c77bfca"
format_stamp: "Formatted at 2026-08-06 04:29:17 on dist-test-slave-ncp9"
server_key: "386e9f502aeede05f091befa8d232d93"
server_key_iv: "d7f452816f64feebe2b25811dfe7e42d"
server_key_version: "encryptionkey@0"
I20260806 04:29:17.093022 29578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260806 04:29:17.095925 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:17.097442 29578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:17.097615 29578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "132e4d711c2e4cd69bbbc10d8c77bfca"
format_stamp: "Formatted at 2026-08-06 04:29:17 on dist-test-slave-ncp9"
server_key: "386e9f502aeede05f091befa8d232d93"
server_key_iv: "d7f452816f64feebe2b25811dfe7e42d"
server_key_version: "encryptionkey@0"
I20260806 04:29:17.097756 29578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:17.116856 29578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:17.120620 29578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:17.120820 29578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:17.121520 29578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:17.122546 29578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:17.122601 29578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:17.122644 29578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:17.122659 29578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:17.134217 29578 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:39527
I20260806 04:29:17.134243 29753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:39527 every 8 connection(s)
I20260806 04:29:17.135226 29578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:17.139365 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29578
I20260806 04:29:17.139478 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:17.139748 29048 external_mini_cluster.cc:1428] Setting key 1244b57a00c4f42fdabb94d0a70907b9
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:17.150004 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:17.137298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:53671 (local address 127.28.94.62:35101)
0806 04:29:17.137453 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:17.137458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:17.138468 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 04:29:17.140036 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:17.140041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:17.140105 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:17.140212 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:17.141705 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.142335 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.143566 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.143791 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.143868 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:17.146450 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:17.146454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:17.146456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:17.146487 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:17.147911 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.148380 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.148382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.148384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.148451 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.148747 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.148749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.148751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.149035 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:17.149134 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:17.149739 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 04:29:17.149859 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:29:17.150964 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:17.137754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35101 (local address 127.28.94.3:53671)
0806 04:29:17.138312 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:17.138363 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:17.139771 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:17.140427 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:17.140437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:17.140878 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:17.141556 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:17.141575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.142472 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.142476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:17.143447 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:17.143459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.143913 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.143916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:17.144065 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:17.144687 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:17.144706 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:17.146267 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:17.148016 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:17.148022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:17.148036 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:17.148307 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:17.148519 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:17.148522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:17.148523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:17.148677 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:17.149223 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:17.149229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:17.149506 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 04:29:17.149701 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:29:17.150005 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260806 04:29:17.152179 29754 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35101
I20260806 04:29:17.152456 29754 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:17.153107 29754 heartbeater.cc:511] Master 127.28.94.62:35101 requested a full tablet report, sending...
I20260806 04:29:17.154491 29120 ts_manager.cc:194] Registered new tserver with Master: 132e4d711c2e4cd69bbbc10d8c77bfca (127.28.94.3:39527)
I20260806 04:29:17.155102 29120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:53671
I20260806 04:29:17.166815 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:17.197003 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:17.186846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.28.94.1:45227)
0806 04:29:17.187204 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:17.187211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:17.187238 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:29:17.187362 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:17.187369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:17.187577 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:17.187720 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:17.188418 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.189306 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.190215 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.190606 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.190700 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:17.193342 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:17.193358 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:17.193376 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:17.193409 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:17.195042 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.195597 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.195602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.195603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.195703 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.195963 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.195965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.195966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.196236 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:17.196397 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:17.196634 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 04:29:17.196787 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":153,"threads_started":1}
W20260806 04:29:17.198362 29265 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53446
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:17.229223 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:17.220661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.28.94.1:45227)
0806 04:29:17.220812 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:17.220817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:17.220930 (+ 113us) server_negotiation.cc:407] Connection header received
0806 04:29:17.221157 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:17.221161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:17.221217 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:17.221290 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:17.222034 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.222577 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.223433 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:17.223674 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:17.223757 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:17.226411 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:17.226414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:17.226416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:17.226449 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:17.227653 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.228075 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.228077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.228078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.228160 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:17.228361 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:17.228364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:17.228365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:17.228594 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:17.228711 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:17.228922 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 04:29:17.229045 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:29:17.230369 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29203
I20260806 04:29:17.246224 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29388
I20260806 04:29:17.270020 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29578
I20260806 04:29:17.286046 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29077
2026-08-06T04:29:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2708 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29774](info): set up 2 sockets
Aug 06 04:29:17 dist-test-slave-ncp9 krb5kdc[29774](info): commencing operation
krb5kdc: starting...
W20260806 04:29:18.402360 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:18 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:18.535888 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39141
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:18.660060 29790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:18.660401 29790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:18.660462 29790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:18.664786 29790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:18.664882 29790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:18.664954 29790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:18.664983 29790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:18.665009 29790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:18.670823 29790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:18.672142 29790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:18.673349 29790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:18.682740 29799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:18.682868 29801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:18.682740 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:18.683127 29790 server_base.cc:1034] running on GCE node
I20260806 04:29:18.683840 29790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:18.685144 29790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:18.686319 29790 hybrid_clock.cc:648] HybridClock initialized: now 1785990558686296 us; error 39 us; skew 500 ppm
Aug 06 04:29:18 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990558, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:18.689227 29790 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:18.690271 29790 webserver.cc:460] Webserver started at http://127.28.94.62:34905/ using document root <none> and password file <none>
I20260806 04:29:18.690922 29790 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:18.690999 29790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:18.691238 29790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:18.693634 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "a6975e4e7ee14b2aaad349fe2059d2a6"
format_stamp: "Formatted at 2026-08-06 04:29:18 on dist-test-slave-ncp9"
server_key: "b1feb372a24f20d38a90ccb9285b40a5"
server_key_iv: "056c122df1effb2b27914d4624c7dba8"
server_key_version: "encryptionkey@0"
I20260806 04:29:18.694267 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "a6975e4e7ee14b2aaad349fe2059d2a6"
format_stamp: "Formatted at 2026-08-06 04:29:18 on dist-test-slave-ncp9"
server_key: "b1feb372a24f20d38a90ccb9285b40a5"
server_key_iv: "056c122df1effb2b27914d4624c7dba8"
server_key_version: "encryptionkey@0"
I20260806 04:29:18.698597 29790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:18.701503 29810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:18.702931 29790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:18.703099 29790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "a6975e4e7ee14b2aaad349fe2059d2a6"
format_stamp: "Formatted at 2026-08-06 04:29:18 on dist-test-slave-ncp9"
server_key: "b1feb372a24f20d38a90ccb9285b40a5"
server_key_iv: "056c122df1effb2b27914d4624c7dba8"
server_key_version: "encryptionkey@0"
I20260806 04:29:18.703233 29790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:18.715579 29790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:18.718933 29790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:18.719139 29790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:18.726841 29790 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39141
I20260806 04:29:18.726857 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39141 every 8 connection(s)
I20260806 04:29:18.727909 29790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:18.731181 29884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:18.734333 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29790
I20260806 04:29:18.734447 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:18.734702 29048 external_mini_cluster.cc:1428] Setting key 9bd4995888650af9a0bae69302716a8f
I20260806 04:29:18.737942 29884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6: Bootstrap starting.
I20260806 04:29:18.740440 29884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:18.741319 29884 log.cc:824] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:18.743530 29884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6: No bootstrap required, opened a new log
Aug 06 04:29:18 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:18.746879 29884 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } }
I20260806 04:29:18.747150 29884 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:18.747215 29884 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6975e4e7ee14b2aaad349fe2059d2a6, State: Initialized, Role: FOLLOWER
I20260806 04:29:18.747712 29884 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } }
I20260806 04:29:18.747851 29884 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:18.747917 29884 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:18.748015 29884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:18.749082 29884 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } }
I20260806 04:29:18.749464 29884 leader_election.cc:302] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6975e4e7ee14b2aaad349fe2059d2a6; no voters:
I20260806 04:29:18.749801 29884 leader_election.cc:288] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:18.749908 29892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:18.750088 29892 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [term 1 LEADER]: Becoming Leader. State: Replica: a6975e4e7ee14b2aaad349fe2059d2a6, State: Running, Role: LEADER
I20260806 04:29:18.750443 29892 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } }
I20260806 04:29:18.750612 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:18.736203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.28.94.62:39141)
0806 04:29:18.736745 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:18.736762 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:18.736802 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:29:18.737519 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:18.737540 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:18.737912 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:18.738271 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:18.739081 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:18.740109 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:18.741069 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:18.741606 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:18.741772 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:18.744417 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:18.744422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:18.744438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:18.744471 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:18.746982 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:18.747516 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:18.747522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:18.747527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:18.747652 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:18.747974 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:18.747977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:18.747979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:18.748470 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:18.748680 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:18.749051 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 04:29:18.749522 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":163,"threads_started":1}
I20260806 04:29:18.751276 29884 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:18.752341 29893 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } } }
I20260806 04:29:18.752481 29893 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:18.752921 29894 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6975e4e7ee14b2aaad349fe2059d2a6. Latest consensus state: current_term: 1 leader_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6975e4e7ee14b2aaad349fe2059d2a6" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39141 } } }
I20260806 04:29:18.753026 29894 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:18.753350 29904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:18.756949 29904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:18.763664 29904 catalog_manager.cc:1347] Generated new cluster ID: a03ede684e2a4aefb8f41bce90f5f96e
I20260806 04:29:18.763775 29904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:18.794946 29904 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:18.796064 29904 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:18.816390 29904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6975e4e7ee14b2aaad349fe2059d2a6: Generated new TSK 0
I20260806 04:29:18.817260 29904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:18.850072 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:18.992131 29919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:18.992450 29919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:18.992497 29919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:18.996673 29919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:18.996759 29919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:18.996824 29919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:19.002496 29919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:19.003693 29919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:19.004789 29919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:19.014948 29930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.014948 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.014968 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:19.015213 29919 server_base.cc:1034] running on GCE node
I20260806 04:29:19.016086 29919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:19.016812 29919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:19.018005 29919 hybrid_clock.cc:648] HybridClock initialized: now 1785990559017975 us; error 41 us; skew 500 ppm
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:19.020998 29919 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:19.021961 29919 webserver.cc:460] Webserver started at http://127.28.94.1:39417/ using document root <none> and password file <none>
I20260806 04:29:19.022552 29919 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:19.022624 29919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:19.022815 29919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:19.025060 29919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "67f4efed73f945a59afa2d9d1765a579"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "1657d48d25c49bdee17091a618f5389c"
server_key_iv: "2fcef8dcb9c4aaa21dc193ce57acc046"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.025648 29919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "67f4efed73f945a59afa2d9d1765a579"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "1657d48d25c49bdee17091a618f5389c"
server_key_iv: "2fcef8dcb9c4aaa21dc193ce57acc046"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.029704 29919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:29:19.032429 29937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.034003 29919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:29:19.034142 29919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "67f4efed73f945a59afa2d9d1765a579"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "1657d48d25c49bdee17091a618f5389c"
server_key_iv: "2fcef8dcb9c4aaa21dc193ce57acc046"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.034258 29919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:19.054484 29919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:19.057590 29919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:19.057806 29919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:19.058459 29919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:19.059417 29919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:19.059471 29919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.059530 29919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:19.059556 29919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.068608 29919 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:43603
I20260806 04:29:19.068624 30082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:43603 every 8 connection(s)
I20260806 04:29:19.069705 29919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:19.079720 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 29919
I20260806 04:29:19.079843 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:19.080147 29048 external_mini_cluster.cc:1428] Setting key 3c7dfea70feeb1f4cb5abb8c32df12b6
I20260806 04:29:19.084843 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.071711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:57865 (local address 127.28.94.62:39141)
0806 04:29:19.071856 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.071861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.072744 (+ 883us) server_negotiation.cc:407] Connection header received
0806 04:29:19.074151 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.074155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.074218 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.074295 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.075565 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.076111 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.077326 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.077556 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.077660 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.080632 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.080636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.080639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.080674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.082505 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.083127 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.083130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.083131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.083206 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.083554 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.083556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.083557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.083761 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.083833 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.084498 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.084670 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:19.085716 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.072089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39141 (local address 127.28.94.1:57865)
0806 04:29:19.072580 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.072628 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.073933 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.074501 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.074508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.074888 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:19.075409 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.075424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.076235 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.076239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.077207 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.077218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.077713 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.077717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.077889 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.078445 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:19.078463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:19.080409 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:19.082633 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.082640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.082660 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.083033 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.083280 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.083282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.083284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.083473 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.083853 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:19.083860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:19.084246 (+ 386us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.084426 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.084734 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260806 04:29:19.086968 30083 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39141
I20260806 04:29:19.087249 30083 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:19.087800 30083 heartbeater.cc:511] Master 127.28.94.62:39141 requested a full tablet report, sending...
I20260806 04:29:19.089500 29830 ts_manager.cc:194] Registered new tserver with Master: 67f4efed73f945a59afa2d9d1765a579 (127.28.94.1:43603)
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:19.090689 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:57865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:19.107905 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:19.228221 30088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:19.228525 30088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:19.228571 30088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:19.233042 30088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:19.233125 30088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:19.233192 30088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:19.239122 30088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:19.240360 30088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:19.241580 30088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:19.252149 30097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.252206 30094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.252168 30095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:19.252489 30088 server_base.cc:1034] running on GCE node
I20260806 04:29:19.253077 30088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:19.253825 30088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:19.254992 30088 hybrid_clock.cc:648] HybridClock initialized: now 1785990559254969 us; error 37 us; skew 500 ppm
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:19.258076 30088 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:19.259096 30088 webserver.cc:460] Webserver started at http://127.28.94.2:40887/ using document root <none> and password file <none>
I20260806 04:29:19.259677 30088 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:19.259738 30088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:19.259928 30088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:19.262174 30088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "a001ff066e8c492f84951770eaaa1c50"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "29bb22df46b5d2dda01e772a5c9c74b6"
server_key_iv: "77d31d8f49696ede1300e6249d1150cf"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.262796 30088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "a001ff066e8c492f84951770eaaa1c50"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "29bb22df46b5d2dda01e772a5c9c74b6"
server_key_iv: "77d31d8f49696ede1300e6249d1150cf"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.266903 30088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:19.269625 30104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.270967 30088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:29:19.271102 30088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "a001ff066e8c492f84951770eaaa1c50"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "29bb22df46b5d2dda01e772a5c9c74b6"
server_key_iv: "77d31d8f49696ede1300e6249d1150cf"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.271220 30088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:19.299808 30088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:19.303136 30088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:19.303335 30088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:19.303990 30088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:19.304941 30088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:19.304991 30088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.305058 30088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:19.305087 30088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.314246 30088 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:33761
I20260806 04:29:19.314276 30264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:33761 every 8 connection(s)
I20260806 04:29:19.315259 30088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:19.315897 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 30088
I20260806 04:29:19.316010 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:19.316272 29048 external_mini_cluster.cc:1428] Setting key 039108f56c9ff8f78a345d0076b65e9c
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:19.334306 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.317462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:44181 (local address 127.28.94.62:39141)
0806 04:29:19.317597 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.317601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.318697 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 04:29:19.320384 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.320388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.320449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.320541 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.321849 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.322406 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.323557 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.323798 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.323889 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.330274 (+ 6385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.330278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.330281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.330317 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.331937 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.332546 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.332550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.332552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.332652 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.332997 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.333000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.333001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.333257 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.333353 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.333942 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.334118 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:19.335130 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.318010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39141 (local address 127.28.94.2:44181)
0806 04:29:19.318551 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.318598 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.320161 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.320731 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.320739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.321165 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:19.321698 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.321713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.322533 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.322535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.323434 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.323445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.323955 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.323960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.324130 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.327549 (+ 3419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:19.327572 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:19.330004 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:19.332098 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.332105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.332125 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.332430 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.332738 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.332740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.332741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.332885 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.333355 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:19.333360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:19.333653 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.333827 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.334127 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":158,"threads_started":1}
I20260806 04:29:19.336433 30265 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39141
I20260806 04:29:19.336760 30265 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:19.337528 30265 heartbeater.cc:511] Master 127.28.94.62:39141 requested a full tablet report, sending...
I20260806 04:29:19.339010 29830 ts_manager.cc:194] Registered new tserver with Master: a001ff066e8c492f84951770eaaa1c50 (127.28.94.2:33761)
I20260806 04:29:19.339648 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:44181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:19.347275 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:19.481552 30271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:19.481870 30271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:19.481918 30271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:19.486330 30271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:19.486413 30271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:19.486477 30271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:19.492288 30271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39141
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:19.493567 30271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:19.494655 30271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:19.504869 30278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.504873 30280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:19.504873 30277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:19.505271 30271 server_base.cc:1034] running on GCE node
I20260806 04:29:19.505738 30271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:19.506451 30271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:19.507604 30271 hybrid_clock.cc:648] HybridClock initialized: now 1785990559507581 us; error 36 us; skew 500 ppm
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:19.510519 30271 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:19.511482 30271 webserver.cc:460] Webserver started at http://127.28.94.3:46747/ using document root <none> and password file <none>
I20260806 04:29:19.512096 30271 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:19.512171 30271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:19.512389 30271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:19.514658 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "c2b2b685ab92493da6bb230163cf3671"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "bc723a0f8963cd465c56ab5fe690bdef"
server_key_iv: "eff482253afdaffa42185b3a0032af93"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.515277 30271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "c2b2b685ab92493da6bb230163cf3671"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "bc723a0f8963cd465c56ab5fe690bdef"
server_key_iv: "eff482253afdaffa42185b3a0032af93"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.519470 30271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:19.522174 30289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.523762 30271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:19.523887 30271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "c2b2b685ab92493da6bb230163cf3671"
format_stamp: "Formatted at 2026-08-06 04:29:19 on dist-test-slave-ncp9"
server_key: "bc723a0f8963cd465c56ab5fe690bdef"
server_key_iv: "eff482253afdaffa42185b3a0032af93"
server_key_version: "encryptionkey@0"
I20260806 04:29:19.523999 30271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:19.553050 30271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:19.556548 30271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:19.556733 30271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:19.557492 30271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:19.558421 30271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:19.558473 30271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.558516 30271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:19.558537 30271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:19.567814 30271 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:43423
I20260806 04:29:19.567867 30441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:43423 every 8 connection(s)
I20260806 04:29:19.568830 30271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:19.577481 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 30271
I20260806 04:29:19.577631 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:19.577926 29048 external_mini_cluster.cc:1428] Setting key 96581025a349e76c767c8175ccba97c5
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:19.583560 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.570928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:50889 (local address 127.28.94.62:39141)
0806 04:29:19.571069 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.571073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.572054 (+ 981us) server_negotiation.cc:407] Connection header received
0806 04:29:19.573469 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.573473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.573533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.573612 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.575007 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.575546 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.576743 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.576999 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.577081 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.580264 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.580268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.580271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.580304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.581538 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.582066 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.582068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.582070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.582142 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.582483 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.582485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.582486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.582682 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.582756 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.583309 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.583428 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:19.584583 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.571334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39141 (local address 127.28.94.3:50889)
0806 04:29:19.571870 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.571921 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.573231 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.573796 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.573804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.574229 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:19.574849 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.574868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.575652 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.575655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.576617 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.576630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.577167 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.577172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.577395 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.578079 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:19.578104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:19.580066 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:19.581659 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.581666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.581684 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.581978 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.582226 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:19.582228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:19.582230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:19.582414 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:19.582785 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:19.582792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:19.583092 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.583278 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.583596 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260806 04:29:19.585901 30443 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39141
I20260806 04:29:19.586179 30443 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:19.586753 30443 heartbeater.cc:511] Master 127.28.94.62:39141 requested a full tablet report, sending...
I20260806 04:29:19.587947 29830 ts_manager.cc:194] Registered new tserver with Master: c2b2b685ab92493da6bb230163cf3671 (127.28.94.3:43423)
I20260806 04:29:19.588543 29830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:50889
I20260806 04:29:19.592260 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:29:19.603060 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.596081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.28.94.62:39141)
0806 04:29:19.596280 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.596285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.596460 (+ 175us) server_negotiation.cc:407] Connection header received
0806 04:29:19.596588 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.596591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.596636 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.596711 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.597531 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.598053 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.598836 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.599036 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.599102 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.600377 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.600381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.600383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.600400 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.601642 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.602030 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.602032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.602034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.602103 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.602303 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.602305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.602306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.602531 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.602635 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.602819 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.602926 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 04:29:19.609035 29830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:19.612589 29830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:19.629933 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.624182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.28.94.1:43603)
0806 04:29:19.624870 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.624876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.624925 (+ 49us) server_negotiation.cc:407] Connection header received
0806 04:29:19.625001 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.625006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.625179 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.625317 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:19.625715 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.626620 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.628426 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629237 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.629596 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.629658 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.629760 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":93,"threads_started":1}
I20260806 04:29:19.629927 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.623503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:43603 (local address 127.0.0.1:48734)
0806 04:29:19.624327 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.624342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.624492 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.625322 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.625327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.625365 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:19.625563 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.625573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.626724 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.626727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.628309 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.628321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629357 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.629477 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.629520 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.629631 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.629780 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":102,"spinlock_wait_cycles":9216,"thread_start_us":147,"threads_started":1}
I20260806 04:29:19.630419 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.623691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:43423 (local address 127.0.0.1:47744)
0806 04:29:19.625498 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.625509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.625593 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.626030 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.626033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.626053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:19.626146 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.626152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.627456 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.627459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.628574 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.628586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629900 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.629982 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.630154 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.630220 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.630291 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754,"mutex_wait_us":46,"thread_start_us":118,"threads_started":1}
I20260806 04:29:19.630571 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.623473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.28.94.2:33761)
0806 04:29:19.623987 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.624000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.624410 (+ 410us) server_negotiation.cc:407] Connection header received
0806 04:29:19.624676 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.624683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.624942 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.625073 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:19.625753 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.626880 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.628467 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629282 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.629598 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.630318 (+ 720us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.630421 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":163,"threads_started":1}
I20260806 04:29:19.631045 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.624197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.28.94.3:43423)
0806 04:29:19.624666 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.624676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.625604 (+ 928us) server_negotiation.cc:407] Connection header received
0806 04:29:19.625697 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.625701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.625883 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.626035 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:19.626257 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.627310 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.628770 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629611 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629714 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.629842 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.630758 (+ 916us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.630871 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":146,"threads_started":1}
I20260806 04:29:19.630833 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.623327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:33761 (local address 127.0.0.1:56792)
0806 04:29:19.624255 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.624316 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.624497 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.625320 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.625326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.625362 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:19.625590 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.625597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.627019 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.627022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.628309 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.628321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.629674 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.629678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.629783 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.629806 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.629873 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.629950 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":118,"spinlock_wait_cycles":4480,"thread_start_us":95,"threads_started":1}
I20260806 04:29:19.633577 29994 tablet_service.cc:1467] Processing CreateTablet for tablet f1e9df6015614651b0b605582d3f7a90 (DEFAULT_TABLE table=test-table [id=e1c1cf8a204f4cbe9157ce50302d23df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:19.634132 30169 tablet_service.cc:1467] Processing CreateTablet for tablet f1e9df6015614651b0b605582d3f7a90 (DEFAULT_TABLE table=test-table [id=e1c1cf8a204f4cbe9157ce50302d23df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:19.634138 30357 tablet_service.cc:1467] Processing CreateTablet for tablet f1e9df6015614651b0b605582d3f7a90 (DEFAULT_TABLE table=test-table [id=e1c1cf8a204f4cbe9157ce50302d23df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:19.634847 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e9df6015614651b0b605582d3f7a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:19.635211 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e9df6015614651b0b605582d3f7a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:19.635211 30169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e9df6015614651b0b605582d3f7a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:19.642856 30462 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Bootstrap starting.
I20260806 04:29:19.643836 30463 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Bootstrap starting.
I20260806 04:29:19.645325 30466 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Bootstrap starting.
I20260806 04:29:19.646035 30463 tablet_bootstrap.cc:654] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:19.646035 30462 tablet_bootstrap.cc:654] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:19.647365 30462 log.cc:824] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:19.647585 30463 log.cc:824] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:19.648061 30466 tablet_bootstrap.cc:654] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:19.648962 30466 log.cc:824] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:19.649892 30462 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: No bootstrap required, opened a new log
I20260806 04:29:19.649891 30463 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: No bootstrap required, opened a new log
I20260806 04:29:19.650107 30463 ts_tablet_manager.cc:1397] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:29:19.650107 30462 ts_tablet_manager.cc:1397] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:29:19.650761 30466 tablet_bootstrap.cc:492] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: No bootstrap required, opened a new log
I20260806 04:29:19.650913 30466 ts_tablet_manager.cc:1397] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:19.654269 30466 raft_consensus.cc:348] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.654644 30466 raft_consensus.cc:374] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:19.654417 30463 raft_consensus.cc:348] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.654706 30466 raft_consensus.cc:729] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a001ff066e8c492f84951770eaaa1c50, State: Initialized, Role: FOLLOWER
I20260806 04:29:19.654733 30463 raft_consensus.cc:374] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:19.654785 30463 raft_consensus.cc:729] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b2b685ab92493da6bb230163cf3671, State: Initialized, Role: FOLLOWER
I20260806 04:29:19.655222 30466 consensus_queue.cc:260] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.655351 30463 consensus_queue.cc:260] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.655917 30462 raft_consensus.cc:348] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.656172 30443 heartbeater.cc:503] Master 127.28.94.62:39141 was elected leader, sending a full tablet report...
I20260806 04:29:19.656203 30462 raft_consensus.cc:374] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:19.656255 30462 raft_consensus.cc:729] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f4efed73f945a59afa2d9d1765a579, State: Initialized, Role: FOLLOWER
I20260806 04:29:19.656385 30463 ts_tablet_manager.cc:1428] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:19.656387 30466 ts_tablet_manager.cc:1428] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:29:19.656728 30265 heartbeater.cc:503] Master 127.28.94.62:39141 was elected leader, sending a full tablet report...
I20260806 04:29:19.656733 30462 consensus_queue.cc:260] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.657382 30083 heartbeater.cc:503] Master 127.28.94.62:39141 was elected leader, sending a full tablet report...
I20260806 04:29:19.657953 30462 ts_tablet_manager.cc:1428] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 04:29:19.755879 30475 raft_consensus.cc:482] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:19.756157 30475 raft_consensus.cc:504] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.757491 30475 leader_election.cc:288] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a001ff066e8c492f84951770eaaa1c50 (127.28.94.2:33761), 67f4efed73f945a59afa2d9d1765a579 (127.28.94.1:43603)
I20260806 04:29:19.761287 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.758016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:50881 (local address 127.28.94.1:43603)
0806 04:29:19.758179 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.758183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.758198 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:19.758255 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.758257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.758298 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.758361 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:19.758645 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.759161 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.760249 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.760682 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.760780 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.760854 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.761088 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.761184 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 04:29:19.761287 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.757827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:43603 (local address 127.28.94.3:50881)
0806 04:29:19.757983 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.757999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.758111 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.758394 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.758397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.758416 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:19.758530 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.758536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.759275 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.759279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.760102 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.760112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.760811 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.760815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.760961 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.760982 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.761050 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.761133 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":21}
I20260806 04:29:19.761560 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.758015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:51159 (local address 127.28.94.2:33761)
0806 04:29:19.758168 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.758173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.758490 (+ 317us) server_negotiation.cc:407] Connection header received
0806 04:29:19.758610 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.758612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.758658 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.758721 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:19.759024 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.759513 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.760560 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.761029 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.761125 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.761200 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.761364 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.761458 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:29:19.761557 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.757874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:33761 (local address 127.28.94.3:51159)
0806 04:29:19.758386 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:19.758398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:19.758505 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:19.758751 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:19.758762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:19.758779 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:19.758910 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.758916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.759621 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.759624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.760430 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:19.760440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.761143 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.761146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:19.761247 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.761269 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:29:19.761334 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:29:19.761405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":125,"thread_start_us":165,"threads_started":1}
I20260806 04:29:19.762254 30019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e9df6015614651b0b605582d3f7a90" candidate_uuid: "c2b2b685ab92493da6bb230163cf3671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f4efed73f945a59afa2d9d1765a579" is_pre_election: true
I20260806 04:29:19.762303 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e9df6015614651b0b605582d3f7a90" candidate_uuid: "c2b2b685ab92493da6bb230163cf3671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a001ff066e8c492f84951770eaaa1c50" is_pre_election: true
I20260806 04:29:19.762668 30209 raft_consensus.cc:2454] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2b2b685ab92493da6bb230163cf3671 in term 0.
I20260806 04:29:19.763253 30290 leader_election.cc:302] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a001ff066e8c492f84951770eaaa1c50, c2b2b685ab92493da6bb230163cf3671; no voters:
I20260806 04:29:19.763556 30475 raft_consensus.cc:2781] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:19.763648 30475 raft_consensus.cc:482] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:19.763687 30475 raft_consensus.cc:3037] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:19.764822 30019 raft_consensus.cc:2454] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2b2b685ab92493da6bb230163cf3671 in term 0.
I20260806 04:29:19.764878 30475 raft_consensus.cc:504] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.765375 30475 leader_election.cc:288] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [CANDIDATE]: Term 1 election: Requested vote from peers a001ff066e8c492f84951770eaaa1c50 (127.28.94.2:33761), 67f4efed73f945a59afa2d9d1765a579 (127.28.94.1:43603)
I20260806 04:29:19.765812 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e9df6015614651b0b605582d3f7a90" candidate_uuid: "c2b2b685ab92493da6bb230163cf3671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a001ff066e8c492f84951770eaaa1c50"
I20260806 04:29:19.765966 30209 raft_consensus.cc:3037] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:19.765955 30019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e9df6015614651b0b605582d3f7a90" candidate_uuid: "c2b2b685ab92493da6bb230163cf3671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f4efed73f945a59afa2d9d1765a579"
I20260806 04:29:19.766095 30019 raft_consensus.cc:3037] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:19.767189 30209 raft_consensus.cc:2454] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2b2b685ab92493da6bb230163cf3671 in term 1.
I20260806 04:29:19.767408 30019 raft_consensus.cc:2454] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2b2b685ab92493da6bb230163cf3671 in term 1.
I20260806 04:29:19.767585 30290 leader_election.cc:302] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a001ff066e8c492f84951770eaaa1c50, c2b2b685ab92493da6bb230163cf3671; no voters:
I20260806 04:29:19.767809 30475 raft_consensus.cc:2781] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:19.768031 30475 raft_consensus.cc:686] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [term 1 LEADER]: Becoming Leader. State: Replica: c2b2b685ab92493da6bb230163cf3671, State: Running, Role: LEADER
I20260806 04:29:19.768405 30475 consensus_queue.cc:237] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } }
I20260806 04:29:19.772269 29829 catalog_manager.cc:5495] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2b2b685ab92493da6bb230163cf3671 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "c2b2b685ab92493da6bb230163cf3671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a001ff066e8c492f84951770eaaa1c50" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f4efed73f945a59afa2d9d1765a579" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2b2b685ab92493da6bb230163cf3671" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 43423 } health_report { overall_health: HEALTHY } } }
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:19.787014 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.778146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.28.94.1:43603)
0806 04:29:19.778283 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.778287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.778575 (+ 288us) server_negotiation.cc:407] Connection header received
0806 04:29:19.778686 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.778689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.778737 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.778807 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.779473 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.779987 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.780852 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.781115 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.781197 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.783775 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.783793 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.783811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.783850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.785427 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.785868 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.785874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.785875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.785970 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.786168 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.786170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.786171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.786446 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.786623 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.786762 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.786878 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
W20260806 04:29:19.817022 30267 tablet.cc:2369] T f1e9df6015614651b0b605582d3f7a90 P a001ff066e8c492f84951770eaaa1c50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:19.817118 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.808558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.28.94.1:43603)
0806 04:29:19.808702 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.808707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.808868 (+ 161us) server_negotiation.cc:407] Connection header received
0806 04:29:19.809078 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.809081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.809132 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.809204 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.809933 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.810470 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.811307 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.811532 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.811635 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.814165 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.814168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.814171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.814202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.815527 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.815970 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.815972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.815973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.816043 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.816237 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.816238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.816239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.816417 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.816510 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.816809 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.816976 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 04:29:19.819984 30444 tablet.cc:2369] T f1e9df6015614651b0b605582d3f7a90 P c2b2b685ab92493da6bb230163cf3671: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:19.821189 30084 tablet.cc:2369] T f1e9df6015614651b0b605582d3f7a90 P 67f4efed73f945a59afa2d9d1765a579: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[29774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:19.846298 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:19.837661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.28.94.1:43603)
0806 04:29:19.837856 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:19.837861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:19.838000 (+ 139us) server_negotiation.cc:407] Connection header received
0806 04:29:19.838159 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:19.838163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:19.838214 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:19.838283 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:19.839052 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.839560 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.840410 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:19.840636 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:19.840730 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:19.843248 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:19.843251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:19.843253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:19.843288 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:19.844588 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.845082 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.845086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.845088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.845177 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:19.845436 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:19.845440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:19.845442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:19.845632 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:19.845748 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:19.846006 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 04:29:19.846135 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 04:29:19.847996 29970 tablet_service.cc:2908] Rejecting scan request for tablet f1e9df6015614651b0b605582d3f7a90: Uninitialized: clean time has not yet been initialized
I20260806 04:29:19.850312 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29919
I20260806 04:29:19.876303 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 30088
I20260806 04:29:19.889768 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 30271
I20260806 04:29:19.901789 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 29790
2026-08-06T04:29:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2612 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[30497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[30497](info): set up 2 sockets
Aug 06 04:29:19 dist-test-slave-ncp9 krb5kdc[30497](info): commencing operation
krb5kdc: starting...
W20260806 04:29:21.023334 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:21.153515 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38809
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:21.275048 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:21.275401 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:21.275460 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:21.279779 30513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:21.279871 30513 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:29:21.279894 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:21.279917 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:21.279939 30513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:21.279958 30513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:21.285928 30513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:38809
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:21.287210 30513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:21.288393 30513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:21.297582 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.297595 30523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.297574 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:21.298027 30513 server_base.cc:1034] running on GCE node
I20260806 04:29:21.298703 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:21.300005 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:21.301167 30513 hybrid_clock.cc:648] HybridClock initialized: now 1785990561301142 us; error 36 us; skew 500 ppm
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:21.304102 30513 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:21.305171 30513 webserver.cc:460] Webserver started at http://127.28.94.62:36417/ using document root <none> and password file <none>
I20260806 04:29:21.305755 30513 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:21.305816 30513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:21.306005 30513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:21.308182 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "7bc135dff30442ee8530a0d79c55379c"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "dd95c30fc66b5eecd8ae17bcef44e1be"
server_key_iv: "73073bf06246f00fecc7de31351a79c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.308781 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "7bc135dff30442ee8530a0d79c55379c"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "dd95c30fc66b5eecd8ae17bcef44e1be"
server_key_iv: "73073bf06246f00fecc7de31351a79c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.312990 30513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:29:21.315797 30531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:21.317207 30513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:21.317340 30513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "7bc135dff30442ee8530a0d79c55379c"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "dd95c30fc66b5eecd8ae17bcef44e1be"
server_key_iv: "73073bf06246f00fecc7de31351a79c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.317451 30513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:21.346731 30513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:21.349858 30513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:21.350036 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:21.358345 30513 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:38809
I20260806 04:29:21.358414 30623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:38809 every 8 connection(s)
I20260806 04:29:21.359370 30513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:21.361519 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 30513
I20260806 04:29:21.361615 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:21.361860 29048 external_mini_cluster.cc:1428] Setting key f7bfe925ec4174c6f2843d96c56ecb94
I20260806 04:29:21.362501 30624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:21.368664 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Bootstrap starting.
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:21.371058 30624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:21.372229 30624 log.cc:824] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:21.374205 30624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: No bootstrap required, opened a new log
I20260806 04:29:21.375241 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.363324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.28.94.62:38809)
0806 04:29:21.363787 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:21.363803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:21.363841 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:29:21.364507 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:21.364524 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:21.364862 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:21.365230 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:21.366026 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.367017 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.367840 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.368310 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.368452 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.370822 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:21.370827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:21.370841 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:21.370869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:21.372471 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.372916 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.372923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.372930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.373047 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.373305 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.373308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.373310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.373722 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:21.373899 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:21.374201 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 04:29:21.374511 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
I20260806 04:29:21.377635 30624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } }
I20260806 04:29:21.377873 30624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:21.377933 30624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc135dff30442ee8530a0d79c55379c, State: Initialized, Role: FOLLOWER
I20260806 04:29:21.378350 30624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } }
I20260806 04:29:21.378482 30624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:21.378554 30624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:21.378654 30624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:21.379578 30624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } }
I20260806 04:29:21.379922 30624 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc135dff30442ee8530a0d79c55379c; no voters:
I20260806 04:29:21.380239 30624 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:21.380350 30630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:21.380573 30630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [term 1 LEADER]: Becoming Leader. State: Replica: 7bc135dff30442ee8530a0d79c55379c, State: Running, Role: LEADER
I20260806 04:29:21.380939 30630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } }
I20260806 04:29:21.381579 30624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:21.382701 30631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc135dff30442ee8530a0d79c55379c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } } }
I20260806 04:29:21.382701 30633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc135dff30442ee8530a0d79c55379c. Latest consensus state: current_term: 1 leader_uuid: "7bc135dff30442ee8530a0d79c55379c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc135dff30442ee8530a0d79c55379c" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38809 } } }
I20260806 04:29:21.382942 30633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:21.382970 30631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:21.383347 30641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:21.386737 30641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:21.393358 30641 catalog_manager.cc:1347] Generated new cluster ID: 6db0c9d653a14a8f98ce013738dc1f19
I20260806 04:29:21.393463 30641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:21.404986 30641 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:21.406430 30641 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:21.418670 30641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Generated new TSK 0
I20260806 04:29:21.419723 30641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:21.452720 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38809
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:21.578086 30655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:21.578408 30655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:21.578454 30655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:21.582926 30655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:21.583010 30655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:21.583052 30655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:21.583096 30655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:21.589030 30655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:21.590376 30655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:21.591506 30655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:21.602469 30663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.602429 30661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.602428 30660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:21.602798 30655 server_base.cc:1034] running on GCE node
I20260806 04:29:21.603266 30655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:21.603986 30655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:21.605150 30655 hybrid_clock.cc:648] HybridClock initialized: now 1785990561605125 us; error 37 us; skew 500 ppm
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:21.608009 30655 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:21.609083 30655 webserver.cc:460] Webserver started at http://127.28.94.1:43373/ using document root <none> and password file <none>
I20260806 04:29:21.609709 30655 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:21.609786 30655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:21.610018 30655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:21.612314 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "c2019d8550db4751a3b65e9aa43467a3"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "609aed6eb2bd272dce26ed83e9a720b3"
server_key_iv: "8902d0e738651ff12d5da41b7809a435"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.613086 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "c2019d8550db4751a3b65e9aa43467a3"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "609aed6eb2bd272dce26ed83e9a720b3"
server_key_iv: "8902d0e738651ff12d5da41b7809a435"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.617242 30655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:21.619963 30673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:21.621392 30655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:21.621551 30655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "c2019d8550db4751a3b65e9aa43467a3"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "609aed6eb2bd272dce26ed83e9a720b3"
server_key_iv: "8902d0e738651ff12d5da41b7809a435"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.621677 30655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:21.636924 30655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:21.640147 30655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:21.640365 30655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:21.642130 30655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:21.642220 30655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:21.642283 30655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:21.642313 30655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:21.652734 30655 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:36771
I20260806 04:29:21.653044 30839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:36771 every 8 connection(s)
I20260806 04:29:21.654006 30655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:21.656813 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.643054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:46171 (local address 127.28.94.62:38809)
0806 04:29:21.643222 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:21.643228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:21.644170 (+ 942us) server_negotiation.cc:407] Connection header received
0806 04:29:21.645442 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:21.645447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:21.645519 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:21.645628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:21.647316 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.648165 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.649561 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.649818 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.649919 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.652784 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:21.652790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:21.652793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:21.652832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:21.654492 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.655152 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.655156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.655159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.655250 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.655595 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.655599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.655600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.655815 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:21.655907 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:21.656465 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 04:29:21.656635 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:29:21.657716 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.643480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.1:46171)
0806 04:29:21.643980 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:21.644034 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:21.645170 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:21.645846 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:21.645857 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:21.646347 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:21.647098 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.647123 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.648332 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.648336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.649417 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.649429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.649984 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.649989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.650196 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.650825 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:21.650848 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:21.652531 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:21.654651 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.654658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.654676 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.655034 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.655338 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.655341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.655343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.655493 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.655923 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:21.655930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:21.656212 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 04:29:21.656396 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 04:29:21.656765 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260806 04:29:21.659168 30840 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38809
I20260806 04:29:21.659341 30840 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:21.659925 30840 heartbeater.cc:511] Master 127.28.94.62:38809 requested a full tablet report, sending...
I20260806 04:29:21.661387 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 30655
I20260806 04:29:21.661509 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:21.661808 29048 external_mini_cluster.cc:1428] Setting key 4ab0c74498970d07e40cc7a9c38d0a99
I20260806 04:29:21.661820 30553 ts_manager.cc:194] Registered new tserver with Master: c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1:36771)
I20260806 04:29:21.663338 30553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:46171
I20260806 04:29:21.669329 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.661622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:37621 (local address 127.28.94.62:38809)
0806 04:29:21.661755 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:21.661760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:21.662156 (+ 396us) server_negotiation.cc:407] Connection header received
0806 04:29:21.662513 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:21.662517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:21.662567 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:21.662655 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:21.663202 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.663909 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.664839 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.665136 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.665225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.666355 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:21.666359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:21.666362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:21.666389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:21.667656 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.668092 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.668095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.668097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.668184 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.668439 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.668442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.668444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.668651 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:21.668739 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:21.669034 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 04:29:21.669157 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260806 04:29:21.669333 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.661773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.1:37621)
0806 04:29:21.662072 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:21.662084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:21.662391 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:21.662703 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:21.662706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:21.662934 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:21.663080 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.663089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.664069 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.664073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.664723 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.664733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.665256 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.665260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.665350 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.665847 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:21.665858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:21.666215 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:21.667772 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.667776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.667778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.667990 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.668267 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.668270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.668272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.668354 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.668741 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:21.668745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:21.668845 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 04:29:21.668961 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 04:29:21.669138 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:21.691345 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38809
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:21.814064 30851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:21.814400 30851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:21.814447 30851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:21.818723 30851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:21.818799 30851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:21.818842 30851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:21.818876 30851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:21.824787 30851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:21.826042 30851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:21.827140 30851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:21.837432 30859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.837451 30860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:21.837436 30863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:21.837831 30851 server_base.cc:1034] running on GCE node
I20260806 04:29:21.838300 30851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:21.839079 30851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:21.840265 30851 hybrid_clock.cc:648] HybridClock initialized: now 1785990561840242 us; error 33 us; skew 500 ppm
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:21.843183 30851 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:21.844143 30851 webserver.cc:460] Webserver started at http://127.28.94.2:45741/ using document root <none> and password file <none>
I20260806 04:29:21.844724 30851 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:21.844789 30851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:21.845029 30851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:21.847222 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "894e0b5bfb479739b039ab5c7643b011"
server_key_iv: "9c543614ae4df8c5e89c41168702e605"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.847803 30851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "894e0b5bfb479739b039ab5c7643b011"
server_key_iv: "9c543614ae4df8c5e89c41168702e605"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.852128 30851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:21.854943 30870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:21.856298 30851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:29:21.856426 30851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
format_stamp: "Formatted at 2026-08-06 04:29:21 on dist-test-slave-ncp9"
server_key: "894e0b5bfb479739b039ab5c7643b011"
server_key_iv: "9c543614ae4df8c5e89c41168702e605"
server_key_version: "encryptionkey@0"
I20260806 04:29:21.856539 30851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:21.871935 30851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:21.875617 30851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:21.875833 30851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:21.877511 30851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:21.877575 30851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:21.877631 30851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:21.877661 30851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:21 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:21.888330 30851 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:39857
I20260806 04:29:21.888379 31022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:39857 every 8 connection(s)
I20260806 04:29:21.889504 30851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:21.890031 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 30851
I20260806 04:29:21.890134 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:21.890453 29048 external_mini_cluster.cc:1428] Setting key a3642171d16dbd139a1381765c699a3b
I20260806 04:29:21.893169 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.878366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:37873 (local address 127.28.94.62:38809)
0806 04:29:21.878488 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:21.878492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:21.879673 (+ 1181us) server_negotiation.cc:407] Connection header received
0806 04:29:21.880933 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:21.880937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:21.881013 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:21.881110 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:21.882777 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.883636 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.885434 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.885796 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.885922 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.888997 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:21.889002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:21.889005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:21.889042 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:21.890840 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.891438 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.891441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.891443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.891535 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.891913 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.891916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.891918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.892153 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:21.892237 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:21.892873 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 04:29:21.893034 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:29:21.894338 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.878975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.2:37873)
0806 04:29:21.879509 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:21.879557 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:21.880610 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:21.881364 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:21.881375 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:21.881795 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:21.882548 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.882573 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.883791 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.883795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.885285 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.885298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.885958 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.885961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.886174 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.886887 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:21.886909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:21.888753 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:21.890989 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.890996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.891013 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.891336 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.891626 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.891628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.891630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.891815 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.892263 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:21.892270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:21.892615 (+ 345us) client_negotiation.cc:769] Sending connection context
0806 04:29:21.892844 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:29:21.893244 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260806 04:29:21.895927 31023 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38809
I20260806 04:29:21.896107 31023 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:21.896741 31023 heartbeater.cc:511] Master 127.28.94.62:38809 requested a full tablet report, sending...
I20260806 04:29:21.898195 30556 ts_manager.cc:194] Registered new tserver with Master: f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857)
I20260806 04:29:21.899020 30556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:37873
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:29:21.907346 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.899367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:45017 (local address 127.28.94.62:38809)
0806 04:29:21.899493 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:21.899499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:21.899514 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:21.899557 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:21.899560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:21.899610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:21.899701 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:21.900334 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.901114 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.902329 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.902570 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.902665 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.903833 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:21.903837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:21.903841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:21.903904 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:21.905465 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.906023 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.906027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.906028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.906132 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:21.906417 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:21.906421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:21.906423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:21.906679 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:21.906776 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:21.907056 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 04:29:21.907182 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:21.907361 31034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:21.898385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.2:45017)
0806 04:29:21.898763 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:21.898782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:21.898887 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:21.899767 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:21.899771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:21.900015 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:21.900185 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.900195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.901264 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.901269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.902187 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:21.902212 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:21.902691 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:21.902695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:21.902771 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:21.903306 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:21.903317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:21.903643 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:21.905580 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.905584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.905586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.905861 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.906227 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:21.906230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:21.906232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:21.906323 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:21.906778 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:21.906782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:21.906884 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 04:29:21.906974 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 04:29:21.907151 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:21.921451 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38809
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:22.044304 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:22.044643 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:22.044698 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:22.048861 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:22.048990 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:22.049041 31036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:22.049081 31036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:22.054711 31036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:22.055876 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:22.056965 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:22.067463 31041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:22.067646 31036 server_base.cc:1034] running on GCE node
W20260806 04:29:22.067466 31042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:22.067878 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:22.068383 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:22.069221 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:22.070426 31036 hybrid_clock.cc:648] HybridClock initialized: now 1785990562070400 us; error 34 us; skew 500 ppm
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:22.073376 31036 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:22.074445 31036 webserver.cc:460] Webserver started at http://127.28.94.3:44247/ using document root <none> and password file <none>
I20260806 04:29:22.075068 31036 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:22.075145 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:22.075362 31036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:22.077692 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "fd44dcaf480e41f1b5b6577f240b9327"
format_stamp: "Formatted at 2026-08-06 04:29:22 on dist-test-slave-ncp9"
server_key: "6940c4de2afd3f9abf7a4c84c097a0a8"
server_key_iv: "fcc4dedfbb2257e6cabb24b3373b00c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:22.078339 31036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "fd44dcaf480e41f1b5b6577f240b9327"
format_stamp: "Formatted at 2026-08-06 04:29:22 on dist-test-slave-ncp9"
server_key: "6940c4de2afd3f9abf7a4c84c097a0a8"
server_key_iv: "fcc4dedfbb2257e6cabb24b3373b00c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:22.082908 31036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:22.085736 31053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:22.087090 31036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:22.087255 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "fd44dcaf480e41f1b5b6577f240b9327"
format_stamp: "Formatted at 2026-08-06 04:29:22 on dist-test-slave-ncp9"
server_key: "6940c4de2afd3f9abf7a4c84c097a0a8"
server_key_iv: "fcc4dedfbb2257e6cabb24b3373b00c7"
server_key_version: "encryptionkey@0"
I20260806 04:29:22.087380 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:22.108633 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:22.111848 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:22.112061 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:22.113907 31036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:22.113970 31036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:22.114030 31036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:22.114076 31036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:22.124729 31036 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:38503
I20260806 04:29:22.124810 31216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:38503 every 8 connection(s)
I20260806 04:29:22.126041 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:22.128538 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.114702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:44895 (local address 127.28.94.62:38809)
0806 04:29:22.114871 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.114877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.116381 (+ 1504us) server_negotiation.cc:407] Connection header received
0806 04:29:22.117775 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.117779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.117847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.117932 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.119479 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.120063 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.121478 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.121714 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.121812 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.124651 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.124665 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.124668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.124706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.126300 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.126941 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.126945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.126946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.127039 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.127386 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.127389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.127391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.127595 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.127696 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.128223 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.128367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:29:22.129489 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.115601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.3:44895)
0806 04:29:22.116185 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.116251 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.117503 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.118168 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.118176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.118624 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:22.119268 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.119293 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.120220 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.120222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.121356 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.121367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.121877 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.121881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.122088 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.122810 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:22.122833 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:22.124417 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:22.126435 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.126441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.126458 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.126840 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.127130 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.127133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.127135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.127288 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.127703 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:22.127710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:22.127978 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.128162 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.128517 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":142,"threads_started":1}
I20260806 04:29:22.129952 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 31036
I20260806 04:29:22.130081 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:22.130345 29048 external_mini_cluster.cc:1428] Setting key 436aeef400d715b0955066aeeabd8a82
I20260806 04:29:22.131001 31217 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38809
I20260806 04:29:22.131168 31217 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:22.131814 31217 heartbeater.cc:511] Master 127.28.94.62:38809 requested a full tablet report, sending...
I20260806 04:29:22.133138 30556 ts_manager.cc:194] Registered new tserver with Master: fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503)
I20260806 04:29:22.133934 30556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:44895
I20260806 04:29:22.139828 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.133272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.28.94.3:54965)
0806 04:29:22.133566 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.133582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.133686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.133904 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.133907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.134131 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:22.134274 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.134281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.135011 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.135015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.135587 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.135596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.136025 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.136029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.136124 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.136573 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:22.136583 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:22.136983 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:22.138301 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.138305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.138307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.138521 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.138795 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.138798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.138800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.138889 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.139261 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:22.139265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:22.139365 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.139453 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.139615 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260806 04:29:22.139822 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.133342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:54965 (local address 127.28.94.62:38809)
0806 04:29:22.133484 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.133489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.133648 (+ 159us) server_negotiation.cc:407] Connection header received
0806 04:29:22.133769 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.133771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.133812 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.133874 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.134386 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.134894 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.135703 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.135924 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.136011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.137110 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.137114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.137117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.137143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.138196 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.138614 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.138618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.138619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.138712 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.138974 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.138977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.138979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.139172 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.139260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.139525 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.139645 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:22.144366 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:22.172247 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.164196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.28.94.62:38809)
0806 04:29:22.164352 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.164356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.164370 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:29:22.164418 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.164420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.164465 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.164541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.165353 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.165889 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.166677 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.166888 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.166964 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.169500 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.169504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.169506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.169556 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.170749 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.171213 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.171215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.171216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.171289 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.171517 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.171519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.171520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.171712 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.171809 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.172005 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.172108 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:29:22.175694 30556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:22.178989 30556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:22.194715 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.188816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36771 (local address 127.0.0.1:45960)
0806 04:29:22.189325 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.189337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.189416 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.190365 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.190368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.190386 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.190493 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.190500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191636 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.191639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.193274 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.193283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.194319 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.194407 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194427 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.194481 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.194550 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":272,"thread_start_us":139,"threads_started":1}
I20260806 04:29:22.194872 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.189714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.28.94.3:38503)
0806 04:29:22.190180 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.190192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.190214 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:29:22.190286 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.190291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.190464 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.190616 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.190783 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191836 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.193274 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.193906 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194212 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.194630 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.194734 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":138,"threads_started":1}
I20260806 04:29:22.194720 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.188438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:39857 (local address 127.0.0.1:45208)
0806 04:29:22.188831 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.188933 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.189077 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.189997 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.190003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.190033 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.190200 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.190209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191831 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.191835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.193128 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.193141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.194271 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.194361 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194385 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.194433 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.194551 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":39,"thread_start_us":99,"threads_started":1}
I20260806 04:29:22.194762 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.189379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.28.94.1:36771)
0806 04:29:22.189908 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.189915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.189937 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:29:22.190016 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.190021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.190220 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.190370 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.190811 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191520 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.193403 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.194181 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194477 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.194532 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.194640 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":97,"threads_started":1}
I20260806 04:29:22.195230 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.188628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.28.94.2:39857)
0806 04:29:22.189407 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.189419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.189445 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:29:22.189533 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.189538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.189805 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.190364 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.190511 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191723 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.193273 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.194120 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194772 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194942 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.194989 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.195073 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":104,"threads_started":1}
I20260806 04:29:22.195381 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.189274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38503 (local address 127.0.0.1:57774)
0806 04:29:22.189572 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.189583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.189675 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.190598 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.190601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.190618 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.190701 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.190706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.191949 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.191952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.193134 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.193145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.194032 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.194035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.194176 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.194224 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.194339 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.194511 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":55,"thread_start_us":113,"threads_started":1}
I20260806 04:29:22.198299 30748 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e (DEFAULT_TABLE table=test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.198343 30934 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e (DEFAULT_TABLE table=test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.198343 31126 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e (DEFAULT_TABLE table=test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.199467 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c0c87ceb4eac84e8b1e5f534553e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.199472 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c0c87ceb4eac84e8b1e5f534553e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.199469 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c0c87ceb4eac84e8b1e5f534553e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.206571 31250 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Bootstrap starting.
I20260806 04:29:22.208308 31252 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Bootstrap starting.
I20260806 04:29:22.208308 31253 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Bootstrap starting.
I20260806 04:29:22.209856 31250 tablet_bootstrap.cc:654] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.210682 31250 log.cc:824] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:22.210732 31252 tablet_bootstrap.cc:654] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.210732 31253 tablet_bootstrap.cc:654] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.211647 31252 log.cc:824] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:22.211647 31253 log.cc:824] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:22.213189 31250 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: No bootstrap required, opened a new log
I20260806 04:29:22.213424 31250 ts_tablet_manager.cc:1397] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 04:29:22.213826 31253 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: No bootstrap required, opened a new log
I20260806 04:29:22.214047 31253 ts_tablet_manager.cc:1397] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:22.214275 31252 tablet_bootstrap.cc:492] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: No bootstrap required, opened a new log
I20260806 04:29:22.214457 31252 ts_tablet_manager.cc:1397] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:22.217938 31250 raft_consensus.cc:348] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.218232 31250 raft_consensus.cc:374] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.218308 31250 raft_consensus.cc:729] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2019d8550db4751a3b65e9aa43467a3, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.218314 31253 raft_consensus.cc:348] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.218390 31252 raft_consensus.cc:348] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.218612 31252 raft_consensus.cc:374] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.218627 31253 raft_consensus.cc:374] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.218685 31253 raft_consensus.cc:729] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6226af9c6a94d31bad9c3dc3c39e496, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.218686 31252 raft_consensus.cc:729] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd44dcaf480e41f1b5b6577f240b9327, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.218853 31250 consensus_queue.cc:260] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.219148 31252 consensus_queue.cc:260] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.219147 31253 consensus_queue.cc:260] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.219732 30840 heartbeater.cc:503] Master 127.28.94.62:38809 was elected leader, sending a full tablet report...
I20260806 04:29:22.220275 31252 ts_tablet_manager.cc:1428] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:29:22.220275 31253 ts_tablet_manager.cc:1428] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:22.220718 31217 heartbeater.cc:503] Master 127.28.94.62:38809 was elected leader, sending a full tablet report...
I20260806 04:29:22.221379 31023 heartbeater.cc:503] Master 127.28.94.62:38809 was elected leader, sending a full tablet report...
I20260806 04:29:22.221489 31250 ts_tablet_manager.cc:1428] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
W20260806 04:29:22.377161 31218 tablet.cc:2369] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:22.391366 31024 tablet.cc:2369] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:22.405675 30841 tablet.cc:2369] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:22.432636 31259 raft_consensus.cc:482] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:22.432873 31259 raft_consensus.cc:504] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.434187 31259 leader_election.cc:288] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857), fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503)
I20260806 04:29:22.439329 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.435185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:43929 (local address 127.28.94.3:38503)
0806 04:29:22.435332 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.435336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.435507 (+ 171us) server_negotiation.cc:407] Connection header received
0806 04:29:22.435611 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.435614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.435673 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.435747 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.435962 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.436617 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.437964 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.438590 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.438709 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.438780 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.439102 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.439182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:29:22.439329 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.434665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:53153 (local address 127.28.94.2:39857)
0806 04:29:22.434814 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.434820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.435016 (+ 196us) server_negotiation.cc:407] Connection header received
0806 04:29:22.435170 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.435174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.435228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.435303 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.435611 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.436374 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.437965 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.438590 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.438707 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.438779 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.439102 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.439191 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:29:22.439402 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.435015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38503 (local address 127.28.94.1:43929)
0806 04:29:22.435416 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.435429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.435528 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.435763 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.435765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.435780 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.435872 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.435877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.436717 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.436720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.437858 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.437865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.438724 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.438728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.438962 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.439005 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.439080 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.439193 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":85,"thread_start_us":108,"threads_started":1}
I20260806 04:29:22.439402 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.434533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:39857 (local address 127.28.94.1:53153)
0806 04:29:22.434894 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.434907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.435051 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.435328 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.435332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.435359 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.435503 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.435513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.436485 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.436488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.437832 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.437847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.438724 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.438728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.438955 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.438999 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.439080 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.439193 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":118,"thread_start_us":156,"threads_started":1}
I20260806 04:29:22.440150 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558c0c87ceb4eac84e8b1e5f534553e" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" is_pre_election: true
I20260806 04:29:22.440215 31152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558c0c87ceb4eac84e8b1e5f534553e" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd44dcaf480e41f1b5b6577f240b9327" is_pre_election: true
I20260806 04:29:22.440511 31152 raft_consensus.cc:2454] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 0.
I20260806 04:29:22.440511 30962 raft_consensus.cc:2454] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 0.
I20260806 04:29:22.441063 30675 leader_election.cc:302] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2019d8550db4751a3b65e9aa43467a3, f6226af9c6a94d31bad9c3dc3c39e496; no voters:
I20260806 04:29:22.441378 31259 raft_consensus.cc:2781] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:22.441469 31259 raft_consensus.cc:482] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:22.441522 31259 raft_consensus.cc:3037] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.442695 31259 raft_consensus.cc:504] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.443076 31259 leader_election.cc:288] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 election: Requested vote from peers f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857), fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503)
I20260806 04:29:22.443528 31152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558c0c87ceb4eac84e8b1e5f534553e" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd44dcaf480e41f1b5b6577f240b9327"
I20260806 04:29:22.443528 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0558c0c87ceb4eac84e8b1e5f534553e" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
I20260806 04:29:22.443663 31152 raft_consensus.cc:3037] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.443677 30962 raft_consensus.cc:3037] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.444774 30962 raft_consensus.cc:2454] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 1.
I20260806 04:29:22.445178 30675 leader_election.cc:302] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2019d8550db4751a3b65e9aa43467a3, f6226af9c6a94d31bad9c3dc3c39e496; no voters:
I20260806 04:29:22.445345 31152 raft_consensus.cc:2454] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 1.
I20260806 04:29:22.445590 31259 raft_consensus.cc:2781] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:22.445863 31259 raft_consensus.cc:686] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 1 LEADER]: Becoming Leader. State: Replica: c2019d8550db4751a3b65e9aa43467a3, State: Running, Role: LEADER
I20260806 04:29:22.446266 31259 consensus_queue.cc:237] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } }
I20260806 04:29:22.449908 30553 catalog_manager.cc:5495] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c2019d8550db4751a3b65e9aa43467a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:22.500118 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.496479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.28.94.1:36771)
0806 04:29:22.496614 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.496618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.496648 (+ 30us) server_negotiation.cc:407] Connection header received
0806 04:29:22.496846 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.496849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.496944 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.497047 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:22.497288 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.497808 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.498676 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.498903 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.498986 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.499276 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.499770 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.499860 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.499925 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.499991 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:22.507321 31152 raft_consensus.cc:1261] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Refusing update from remote peer c2019d8550db4751a3b65e9aa43467a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.508142 31259 consensus_queue.cc:1035] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.509186 30962 raft_consensus.cc:1261] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Refusing update from remote peer c2019d8550db4751a3b65e9aa43467a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.509858 31259 consensus_queue.cc:1035] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.518656 31270 mvcc.cc:205] Tried to move back new op lower bound from 7315417344020672512 to 7315417343783444480. Current Snapshot: MvccSnapshot[applied={T|T < 7315417344020672512}]
I20260806 04:29:22.520635 31272 mvcc.cc:205] Tried to move back new op lower bound from 7315417344020672512 to 7315417343783444480. Current Snapshot: MvccSnapshot[applied={T|T < 7315417344020672512}]
I20260806 04:29:22.530689 30553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:22.530946 30553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:22.533972 30553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Sending DeleteTablet for 3 replicas of tablet 0558c0c87ceb4eac84e8b1e5f534553e
I20260806 04:29:22.535006 30934 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45208
I20260806 04:29:22.535017 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260806 04:29:22.535691 31284 tablet_replica.cc:330] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: stopping tablet replica
I20260806 04:29:22.535817 30748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0558c0c87ceb4eac84e8b1e5f534553e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45960
I20260806 04:29:22.535967 30553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:29:22.536096 31284 raft_consensus.cc:2229] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 04:29:22.536456 30553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:22.536470 31284 raft_consensus.cc:2258] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.536911 31283 tablet_replica.cc:330] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: stopping tablet replica
I20260806 04:29:22.537247 31283 raft_consensus.cc:2229] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:22.537572 31283 raft_consensus.cc:2258] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.537882 31285 tablet_replica.cc:330] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: stopping tablet replica
I20260806 04:29:22.538311 31285 raft_consensus.cc:2229] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:22.538524 31284 ts_tablet_manager.cc:1905] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.538731 31285 raft_consensus.cc:2258] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.542672 30748 tablet_service.cc:1467] Processing CreateTablet for tablet bc222fec344a4993b497c2741472461f (DEFAULT_TABLE table=test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.542936 31284 ts_tablet_manager.cc:1918] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:22.543006 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc222fec344a4993b497c2741472461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.543051 30934 tablet_service.cc:1467] Processing CreateTablet for tablet bc222fec344a4993b497c2741472461f (DEFAULT_TABLE table=test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.543221 31284 log.cc:1196] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/0558c0c87ceb4eac84e8b1e5f534553e
I20260806 04:29:22.543322 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc222fec344a4993b497c2741472461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.543715 31284 ts_tablet_manager.cc:1939] T 0558c0c87ceb4eac84e8b1e5f534553e P fd44dcaf480e41f1b5b6577f240b9327: Deleting consensus metadata
I20260806 04:29:22.544461 31283 ts_tablet_manager.cc:1905] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.545334 30532 catalog_manager.cc:4841] TS fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503): tablet 0558c0c87ceb4eac84e8b1e5f534553e (table test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]) successfully deleted
I20260806 04:29:22.545711 31253 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Bootstrap starting.
I20260806 04:29:22.545955 31126 tablet_service.cc:1467] Processing CreateTablet for tablet bc222fec344a4993b497c2741472461f (DEFAULT_TABLE table=test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:22.546203 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc222fec344a4993b497c2741472461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.546772 31253 tablet_bootstrap.cc:654] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.548333 31283 ts_tablet_manager.cc:1918] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:22.548431 31283 log.cc:1196] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/0558c0c87ceb4eac84e8b1e5f534553e
I20260806 04:29:22.548604 31252 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Bootstrap starting.
I20260806 04:29:22.548749 31250 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Bootstrap starting.
I20260806 04:29:22.549610 31252 tablet_bootstrap.cc:654] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.550035 31250 tablet_bootstrap.cc:654] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.551082 31252 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: No bootstrap required, opened a new log
I20260806 04:29:22.551186 31252 ts_tablet_manager.cc:1397] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:22.551676 31283 ts_tablet_manager.cc:1939] T 0558c0c87ceb4eac84e8b1e5f534553e P f6226af9c6a94d31bad9c3dc3c39e496: Deleting consensus metadata
I20260806 04:29:22.551731 31252 raft_consensus.cc:348] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.551844 31252 raft_consensus.cc:374] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.551874 31252 raft_consensus.cc:729] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd44dcaf480e41f1b5b6577f240b9327, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.551868 31285 ts_tablet_manager.cc:1905] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.552014 31252 consensus_queue.cc:260] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.552285 31252 ts_tablet_manager.cc:1428] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:22.553836 30532 catalog_manager.cc:4841] TS f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857): tablet 0558c0c87ceb4eac84e8b1e5f534553e (table test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]) successfully deleted
I20260806 04:29:22.554925 31285 ts_tablet_manager.cc:1918] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:22.555042 31285 log.cc:1196] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/0558c0c87ceb4eac84e8b1e5f534553e
I20260806 04:29:22.555371 31285 ts_tablet_manager.cc:1939] T 0558c0c87ceb4eac84e8b1e5f534553e P c2019d8550db4751a3b65e9aa43467a3: Deleting consensus metadata
I20260806 04:29:22.556398 31250 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: No bootstrap required, opened a new log
I20260806 04:29:22.556502 31250 ts_tablet_manager.cc:1397] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 04:29:22.556645 30535 catalog_manager.cc:4841] TS c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1:36771): tablet 0558c0c87ceb4eac84e8b1e5f534553e (table test-table [id=9fbc6534ead349ee9e0b52be6fbc9f82]) successfully deleted
I20260806 04:29:22.556816 31253 tablet_bootstrap.cc:492] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: No bootstrap required, opened a new log
I20260806 04:29:22.556933 31253 ts_tablet_manager.cc:1397] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260806 04:29:22.557119 31250 raft_consensus.cc:348] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.557258 31250 raft_consensus.cc:374] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.557301 31250 raft_consensus.cc:729] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2019d8550db4751a3b65e9aa43467a3, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.557439 31250 consensus_queue.cc:260] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.557433 31253 raft_consensus.cc:348] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.557538 31253 raft_consensus.cc:374] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.557574 31253 raft_consensus.cc:729] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6226af9c6a94d31bad9c3dc3c39e496, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.557749 31250 ts_tablet_manager.cc:1428] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:22.557710 31253 consensus_queue.cc:260] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.557957 31253 ts_tablet_manager.cc:1428] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:22.659415 31275 raft_consensus.cc:482] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:22.659596 31275 raft_consensus.cc:504] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.660185 31275 leader_election.cc:288] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857), fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503)
I20260806 04:29:22.660717 31152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc222fec344a4993b497c2741472461f" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd44dcaf480e41f1b5b6577f240b9327" is_pre_election: true
I20260806 04:29:22.660796 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc222fec344a4993b497c2741472461f" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" is_pre_election: true
I20260806 04:29:22.660882 31152 raft_consensus.cc:2454] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 0.
I20260806 04:29:22.660984 30962 raft_consensus.cc:2454] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 0.
I20260806 04:29:22.661293 30675 leader_election.cc:302] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2019d8550db4751a3b65e9aa43467a3, fd44dcaf480e41f1b5b6577f240b9327; no voters:
I20260806 04:29:22.661549 31275 raft_consensus.cc:2781] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:22.661617 31275 raft_consensus.cc:482] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:22.661669 31275 raft_consensus.cc:3037] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.663069 31275 raft_consensus.cc:504] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.663506 31275 leader_election.cc:288] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 election: Requested vote from peers f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857), fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503)
I20260806 04:29:22.663949 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc222fec344a4993b497c2741472461f" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
I20260806 04:29:22.663959 31152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc222fec344a4993b497c2741472461f" candidate_uuid: "c2019d8550db4751a3b65e9aa43467a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd44dcaf480e41f1b5b6577f240b9327"
I20260806 04:29:22.664083 30962 raft_consensus.cc:3037] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.664091 31152 raft_consensus.cc:3037] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.665335 31152 raft_consensus.cc:2454] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 1.
I20260806 04:29:22.665720 30962 raft_consensus.cc:2454] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2019d8550db4751a3b65e9aa43467a3 in term 1.
I20260806 04:29:22.665838 30675 leader_election.cc:302] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2019d8550db4751a3b65e9aa43467a3, fd44dcaf480e41f1b5b6577f240b9327; no voters:
I20260806 04:29:22.666057 31275 raft_consensus.cc:2781] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:22.666180 31275 raft_consensus.cc:686] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 1 LEADER]: Becoming Leader. State: Replica: c2019d8550db4751a3b65e9aa43467a3, State: Running, Role: LEADER
I20260806 04:29:22.666332 31275 consensus_queue.cc:237] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.668370 30550 catalog_manager.cc:5495] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c2019d8550db4751a3b65e9aa43467a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990561, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:22.731451 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.722939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38809 (local address 127.0.0.1:47088)
0806 04:29:22.723296 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.723312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.723425 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.723671 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.723673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.724128 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:22.724255 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.724263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.725082 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.725084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.725649 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.725658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.726055 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.726058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.726122 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.726747 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:22.726760 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:22.728233 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:22.729899 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.729911 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.729918 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.730196 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.730410 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:22.730411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:22.730413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:22.730504 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:22.730861 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:22.730867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:22.730990 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.731096 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.731261 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260806 04:29:22.731467 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.723015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.28.94.62:38809)
0806 04:29:22.723329 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.723336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.723371 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:29:22.723506 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.723508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.723553 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.723627 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.724363 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.724989 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.725742 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.725966 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.726046 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.728396 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.728399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.728402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.728426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.729767 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.730274 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.730277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.730278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.730348 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.730582 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.730584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.730585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.730777 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.730852 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.731151 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.731325 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":247,"thread_start_us":140,"threads_started":1}
I20260806 04:29:22.735787 30550 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:47088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:29:22.736542 30550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:22.741721 30748 tablet_service.cc:1467] Processing CreateTablet for tablet da94d6571a2e4078ae2196d96dd66bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43b1cb8f5c76453bafbb0d3854edd85b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:22.741803 31126 tablet_service.cc:1467] Processing CreateTablet for tablet da94d6571a2e4078ae2196d96dd66bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43b1cb8f5c76453bafbb0d3854edd85b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:22.741802 30934 tablet_service.cc:1467] Processing CreateTablet for tablet da94d6571a2e4078ae2196d96dd66bfa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43b1cb8f5c76453bafbb0d3854edd85b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:22.742225 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da94d6571a2e4078ae2196d96dd66bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.742225 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da94d6571a2e4078ae2196d96dd66bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.742238 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da94d6571a2e4078ae2196d96dd66bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:22.744942 31253 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: Bootstrap starting.
I20260806 04:29:22.745186 31252 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: Bootstrap starting.
I20260806 04:29:22.745312 31250 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: Bootstrap starting.
I20260806 04:29:22.746111 31253 tablet_bootstrap.cc:654] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.746155 31250 tablet_bootstrap.cc:654] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.746165 31252 tablet_bootstrap.cc:654] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:22.747479 31250 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: No bootstrap required, opened a new log
I20260806 04:29:22.747546 31252 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: No bootstrap required, opened a new log
I20260806 04:29:22.747586 31250 ts_tablet_manager.cc:1397] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:29:22.747645 31252 ts_tablet_manager.cc:1397] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:22.747831 31253 tablet_bootstrap.cc:492] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: No bootstrap required, opened a new log
I20260806 04:29:22.747921 31253 ts_tablet_manager.cc:1397] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:22.748077 31250 raft_consensus.cc:348] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748207 31250 raft_consensus.cc:374] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.748148 31252 raft_consensus.cc:348] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748242 31250 raft_consensus.cc:729] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2019d8550db4751a3b65e9aa43467a3, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.748252 31252 raft_consensus.cc:374] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.748283 31252 raft_consensus.cc:729] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd44dcaf480e41f1b5b6577f240b9327, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.748364 31253 raft_consensus.cc:348] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748377 31250 consensus_queue.cc:260] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748402 31252 consensus_queue.cc:260] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748473 31253 raft_consensus.cc:374] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:22.748510 31253 raft_consensus.cc:729] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6226af9c6a94d31bad9c3dc3c39e496, State: Initialized, Role: FOLLOWER
I20260806 04:29:22.748674 31252 ts_tablet_manager.cc:1428] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:22.748632 31253 consensus_queue.cc:260] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.748682 31250 ts_tablet_manager.cc:1428] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:22.749023 31260 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.749187 31260 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.748878 31253 ts_tablet_manager.cc:1428] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:22.749488 31275 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.749421 31262 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.749642 31262 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.749635 31275 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.759513 31260 raft_consensus.cc:482] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:22.759686 31260 raft_consensus.cc:504] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.760960 31260 leader_election.cc:288] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1:36771), f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857)
I20260806 04:29:22.766469 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.761305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:38799 (local address 127.28.94.1:36771)
0806 04:29:22.761465 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.761470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.761590 (+ 120us) server_negotiation.cc:407] Connection header received
0806 04:29:22.761753 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.761757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.761812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.761895 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.762475 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.763235 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765165 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.765785 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.765969 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.766247 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.766328 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:29:22.766469 31245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.761574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:51721 (local address 127.28.94.2:39857)
0806 04:29:22.761733 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.761737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.761748 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:29:22.761795 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.761798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.761839 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.761910 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:22.762384 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.763209 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765157 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.765786 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765898 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.765967 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.766247 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.766325 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:29:22.766547 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.761176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:39857 (local address 127.28.94.3:51721)
0806 04:29:22.761582 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.761594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.761659 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.761923 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.761926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.761973 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.762255 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.762265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.763319 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.763323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.765020 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.765035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.765905 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.766099 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.766158 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.766219 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.766335 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":470,"spinlock_wait_cycles":83968,"thread_start_us":65,"threads_started":1}
I20260806 04:29:22.766891 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.761175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36771 (local address 127.28.94.3:38799)
0806 04:29:22.761492 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.761506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.761644 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.761923 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.761926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.761971 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:22.762361 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.762370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.763319 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.763323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.765020 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.765035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.765905 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.765909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.766082 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.766139 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.766220 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.766335 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":185,"spinlock_wait_cycles":38272,"thread_start_us":111,"threads_started":1}
I20260806 04:29:22.767100 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da94d6571a2e4078ae2196d96dd66bfa" candidate_uuid: "fd44dcaf480e41f1b5b6577f240b9327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" is_pre_election: true
I20260806 04:29:22.767280 30962 raft_consensus.cc:2454] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd44dcaf480e41f1b5b6577f240b9327 in term 0.
I20260806 04:29:22.767534 30778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da94d6571a2e4078ae2196d96dd66bfa" candidate_uuid: "fd44dcaf480e41f1b5b6577f240b9327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2019d8550db4751a3b65e9aa43467a3" is_pre_election: true
I20260806 04:29:22.767711 31054 leader_election.cc:302] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6226af9c6a94d31bad9c3dc3c39e496, fd44dcaf480e41f1b5b6577f240b9327; no voters:
I20260806 04:29:22.767802 30778 raft_consensus.cc:2454] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd44dcaf480e41f1b5b6577f240b9327 in term 0.
I20260806 04:29:22.768006 31260 raft_consensus.cc:2781] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:22.768071 31260 raft_consensus.cc:482] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:22.768115 31260 raft_consensus.cc:3037] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.769290 31260 raft_consensus.cc:504] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.769658 31260 leader_election.cc:288] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [CANDIDATE]: Term 1 election: Requested vote from peers c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1:36771), f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857)
I20260806 04:29:22.770008 30778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da94d6571a2e4078ae2196d96dd66bfa" candidate_uuid: "fd44dcaf480e41f1b5b6577f240b9327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2019d8550db4751a3b65e9aa43467a3"
I20260806 04:29:22.770061 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da94d6571a2e4078ae2196d96dd66bfa" candidate_uuid: "fd44dcaf480e41f1b5b6577f240b9327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6226af9c6a94d31bad9c3dc3c39e496"
I20260806 04:29:22.770144 30778 raft_consensus.cc:3037] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.770197 30962 raft_consensus.cc:3037] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:22.771235 30962 raft_consensus.cc:2454] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd44dcaf480e41f1b5b6577f240b9327 in term 1.
I20260806 04:29:22.771275 30778 raft_consensus.cc:2454] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd44dcaf480e41f1b5b6577f240b9327 in term 1.
I20260806 04:29:22.771529 31054 leader_election.cc:302] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6226af9c6a94d31bad9c3dc3c39e496, fd44dcaf480e41f1b5b6577f240b9327; no voters:
I20260806 04:29:22.771735 31260 raft_consensus.cc:2781] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:22.771937 31260 raft_consensus.cc:686] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [term 1 LEADER]: Becoming Leader. State: Replica: fd44dcaf480e41f1b5b6577f240b9327, State: Running, Role: LEADER
I20260806 04:29:22.772190 31260 consensus_queue.cc:237] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } }
I20260806 04:29:22.773595 31306 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: TxnStatusTablet state changed. Reason: New leader fd44dcaf480e41f1b5b6577f240b9327. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.773723 31306 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:22.774062 31310 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:29:22.774240 31310 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:29:22.774997 30556 catalog_manager.cc:5495] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:22.788041 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.783893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.28.94.3:38503)
0806 04:29:22.784012 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.784016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.784126 (+ 110us) server_negotiation.cc:407] Connection header received
0806 04:29:22.784334 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.784336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.784392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.784474 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:22.784763 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.785320 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.786536 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.786746 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.786814 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.787165 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.787635 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.787731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.787855 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.787914 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:29:22.788091 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.783787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38503 (local address 127.0.0.1:57778)
0806 04:29:22.784052 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.784069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.784198 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.784517 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.784521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.784532 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:22.784666 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.784674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.785430 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.785434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.786420 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.786433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.786848 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.786852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.786948 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.787018 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.787739 (+ 721us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.787758 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.787833 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.787938 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260806 04:29:22.791618 30962 raft_consensus.cc:1261] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Refusing update from remote peer fd44dcaf480e41f1b5b6577f240b9327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.791857 30778 raft_consensus.cc:1261] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Refusing update from remote peer fd44dcaf480e41f1b5b6577f240b9327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.792171 31306 consensus_queue.cc:1035] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.792448 31260 consensus_queue.cc:1035] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.794790 31275 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: TxnStatusTablet state changed. Reason: New leader fd44dcaf480e41f1b5b6577f240b9327. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.794934 31275 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.795218 31262 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: TxnStatusTablet state changed. Reason: New leader fd44dcaf480e41f1b5b6577f240b9327. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.795336 31262 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.798050 31306 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.798220 31306 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:22.798357 31306 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.798431 31306 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P fd44dcaf480e41f1b5b6577f240b9327: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:22.798666 31269 mvcc.cc:205] Tried to move back new op lower bound from 7315417345188810752 to 7315417345117507584. Current Snapshot: MvccSnapshot[applied={T|T < 7315417345188810752}]
I20260806 04:29:22.798794 31262 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.798905 31262 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P f6226af9c6a94d31bad9c3dc3c39e496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.798852 31275 tablet_replica.cc:439] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd44dcaf480e41f1b5b6577f240b9327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2019d8550db4751a3b65e9aa43467a3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36771 } } peers { permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 } } peers { permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 } } }
I20260806 04:29:22.798952 31275 tablet_replica.cc:442] T da94d6571a2e4078ae2196d96dd66bfa P c2019d8550db4751a3b65e9aa43467a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:22.815510 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.811908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38503 (local address 127.28.94.1:43749)
0806 04:29:22.812305 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.812318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.812569 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.812827 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.812830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.812839 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:22.812973 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.812979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.813693 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.813697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.814332 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.814341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.814775 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.814779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.814859 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.814911 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.815205 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.815212 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.815271 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.815366 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":136,"threads_started":1}
I20260806 04:29:22.815497 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.812043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:43749 (local address 127.28.94.3:38503)
0806 04:29:22.812178 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.812183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.812515 (+ 332us) server_negotiation.cc:407] Connection header received
0806 04:29:22.812689 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.812691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.812742 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.812803 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:22.813084 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.813584 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.814448 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.814668 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.814764 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.815051 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.815131 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.815190 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.815288 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.815344 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:29:22.823504 31152 raft_consensus.cc:1261] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Refusing update from remote peer c2019d8550db4751a3b65e9aa43467a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.823616 30962 raft_consensus.cc:1261] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Refusing update from remote peer c2019d8550db4751a3b65e9aa43467a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:22.824014 31265 consensus_queue.cc:1035] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6226af9c6a94d31bad9c3dc3c39e496" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.824193 31275 consensus_queue.cc:1035] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd44dcaf480e41f1b5b6577f240b9327" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:22.850838 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.846443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:54075 (local address 127.28.94.1:36771)
0806 04:29:22.846569 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.846573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.846795 (+ 222us) server_negotiation.cc:407] Connection header received
0806 04:29:22.846956 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.846959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.847021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.847099 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:22.847713 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.848372 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.849635 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.849862 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.849939 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.850278 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.850377 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.850440 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.850614 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.850694 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:29:22.850869 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.846312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36771 (local address 127.28.94.3:54075)
0806 04:29:22.846697 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:22.846713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:22.846836 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:22.847416 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:22.847420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:22.847431 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:22.847595 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.847603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.848481 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.848485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.849522 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:22.849534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.849961 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.849964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:22.850063 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.850130 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:22.850461 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:22.850469 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:29:22.850568 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:29:22.850661 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260806 04:29:22.876703 30556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:22.876950 30556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:22.879355 30556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bc135dff30442ee8530a0d79c55379c: Sending DeleteTablet for 3 replicas of tablet bc222fec344a4993b497c2741472461f
I20260806 04:29:22.880149 30748 tablet_service.cc:1514] Processing DeleteTablet for tablet bc222fec344a4993b497c2741472461f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45960
I20260806 04:29:22.880157 30934 tablet_service.cc:1514] Processing DeleteTablet for tablet bc222fec344a4993b497c2741472461f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45208
I20260806 04:29:22.880367 31283 tablet_replica.cc:330] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: stopping tablet replica
I20260806 04:29:22.880367 31285 tablet_replica.cc:330] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: stopping tablet replica
I20260806 04:29:22.880498 31285 raft_consensus.cc:2229] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:22.880496 31283 raft_consensus.cc:2229] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:22.880633 31283 raft_consensus.cc:2258] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.880719 31285 raft_consensus.cc:2258] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.880856 31126 tablet_service.cc:1514] Processing DeleteTablet for tablet bc222fec344a4993b497c2741472461f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:57774
I20260806 04:29:22.881088 31284 tablet_replica.cc:330] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: stopping tablet replica
I20260806 04:29:22.881206 31284 raft_consensus.cc:2229] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:22.881327 31284 raft_consensus.cc:2258] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:22.881650 31285 ts_tablet_manager.cc:1905] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.882055 31284 ts_tablet_manager.cc:1905] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.882175 31283 ts_tablet_manager.cc:1905] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:22.885042 31283 ts_tablet_manager.cc:1918] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:22.885042 31285 ts_tablet_manager.cc:1918] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:22.885133 31283 log.cc:1196] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/bc222fec344a4993b497c2741472461f
I20260806 04:29:22.885138 31285 log.cc:1196] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/bc222fec344a4993b497c2741472461f
I20260806 04:29:22.885250 31284 ts_tablet_manager.cc:1918] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:22.885396 31284 log.cc:1196] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/bc222fec344a4993b497c2741472461f
I20260806 04:29:22.885399 31283 ts_tablet_manager.cc:1939] T bc222fec344a4993b497c2741472461f P f6226af9c6a94d31bad9c3dc3c39e496: Deleting consensus metadata
I20260806 04:29:22.885406 31285 ts_tablet_manager.cc:1939] T bc222fec344a4993b497c2741472461f P c2019d8550db4751a3b65e9aa43467a3: Deleting consensus metadata
I20260806 04:29:22.885684 31284 ts_tablet_manager.cc:1939] T bc222fec344a4993b497c2741472461f P fd44dcaf480e41f1b5b6577f240b9327: Deleting consensus metadata
I20260806 04:29:22.886307 30535 catalog_manager.cc:4841] TS c2019d8550db4751a3b65e9aa43467a3 (127.28.94.1:36771): tablet bc222fec344a4993b497c2741472461f (table test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]) successfully deleted
I20260806 04:29:22.886418 30532 catalog_manager.cc:4841] TS fd44dcaf480e41f1b5b6577f240b9327 (127.28.94.3:38503): tablet bc222fec344a4993b497c2741472461f (table test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]) successfully deleted
I20260806 04:29:22.887569 30532 catalog_manager.cc:4841] TS f6226af9c6a94d31bad9c3dc3c39e496 (127.28.94.2:39857): tablet bc222fec344a4993b497c2741472461f (table test-table [id=b0c5c579740d4eb9b927e6d5b1d74d43]) successfully deleted
Aug 06 04:29:22 dist-test-slave-ncp9 krb5kdc[30497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:22.900995 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.887849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.28.94.1:36771)
0806 04:29:22.887995 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.888001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.888139 (+ 138us) server_negotiation.cc:407] Connection header received
0806 04:29:22.888353 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.888356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.888415 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.888503 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.889423 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.890228 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.891379 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.891753 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.891851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.896582 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.896604 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.896626 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.896668 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.898874 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.899430 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.899438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.899440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.899551 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.899807 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.899810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.899812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.900185 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.900410 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.900630 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.900779 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:29:22.901561 30832 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984
I20260806 04:29:22.912851 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:22.903744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.28.94.62:38809)
0806 04:29:22.903900 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:22.903905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:22.904116 (+ 211us) server_negotiation.cc:407] Connection header received
0806 04:29:22.904236 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:22.904239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:22.904294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:22.904385 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:22.905195 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.906043 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.907208 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:22.907552 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:22.907641 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:22.909624 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:22.909628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:22.909631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:22.909666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:22.911076 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.911541 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.911543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.911544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.911623 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:22.911894 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:22.911896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:22.911897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:22.912091 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:22.912189 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:22.912536 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 04:29:22.912676 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 04:29:22.913326 30556 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47090
I20260806 04:29:22.914814 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 30655
I20260806 04:29:22.930729 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 30851
I20260806 04:29:22.949795 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 31036
I20260806 04:29:22.963548 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 30513
2026-08-06T04:29:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3067 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:23 dist-test-slave-ncp9 krb5kdc[31335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:23 dist-test-slave-ncp9 krb5kdc[31335](info): set up 2 sockets
Aug 06 04:29:23 dist-test-slave-ncp9 krb5kdc[31335](info): commencing operation
krb5kdc: starting...
W20260806 04:29:24.090149 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:24.221820 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46843
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:24.344188 31350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:24.344503 31350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:24.344545 31350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:24.348806 31350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:24.348915 31350 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:29:24.348944 31350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:24.348977 31350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:24.349005 31350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:24.349031 31350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:24.354823 31350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46843
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:24.356081 31350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:24.357237 31350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:24.366541 31361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.366541 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.366554 31359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:24.366866 31350 server_base.cc:1034] running on GCE node
I20260806 04:29:24.367621 31350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:24.368867 31350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:24.370069 31350 hybrid_clock.cc:648] HybridClock initialized: now 1785990564370051 us; error 32 us; skew 500 ppm
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:24.377051 31350 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:24.378271 31350 webserver.cc:460] Webserver started at http://127.28.94.62:41067/ using document root <none> and password file <none>
I20260806 04:29:24.379086 31350 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:24.379190 31350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:24.379460 31350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:24.382730 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5a6eb272ebff41fb8b0b8b63202b887d"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "2b302efc1dd080a10764433fe4ebff19"
server_key_iv: "60a4b56603c021eced3502b77e7ab364"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.383529 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5a6eb272ebff41fb8b0b8b63202b887d"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "2b302efc1dd080a10764433fe4ebff19"
server_key_iv: "60a4b56603c021eced3502b77e7ab364"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.391101 31350 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260806 04:29:24.394934 31370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:24.396520 31350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:24.396693 31350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5a6eb272ebff41fb8b0b8b63202b887d"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "2b302efc1dd080a10764433fe4ebff19"
server_key_iv: "60a4b56603c021eced3502b77e7ab364"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.396848 31350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:24.427388 31350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:24.431874 31350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:24.432116 31350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:24.440549 31350 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46843
I20260806 04:29:24.440572 31456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46843 every 8 connection(s)
I20260806 04:29:24.441696 31350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:24.444926 31457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:24.450639 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 31350
I20260806 04:29:24.450753 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:24.451001 29048 external_mini_cluster.cc:1428] Setting key 011a04d637faaa8b2d4e6915cec1d533
I20260806 04:29:24.451297 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Bootstrap starting.
I20260806 04:29:24.454151 31457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:24.454984 31457 log.cc:824] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:24.457177 31457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: No bootstrap required, opened a new log
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:24.461369 31457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } }
I20260806 04:29:24.461701 31457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:24.461779 31457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6eb272ebff41fb8b0b8b63202b887d, State: Initialized, Role: FOLLOWER
I20260806 04:29:24.462339 31457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } }
I20260806 04:29:24.462492 31457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:24.462564 31457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:24.462667 31457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:24.463888 31457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } }
I20260806 04:29:24.464269 31457 leader_election.cc:302] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a6eb272ebff41fb8b0b8b63202b887d; no voters:
I20260806 04:29:24.464684 31457 leader_election.cc:288] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:24.464790 31463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:24.465057 31463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [term 1 LEADER]: Becoming Leader. State: Replica: 5a6eb272ebff41fb8b0b8b63202b887d, State: Running, Role: LEADER
I20260806 04:29:24.465404 31463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } }
I20260806 04:29:24.466113 31457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:24.466326 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.452534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.28.94.62:46843)
0806 04:29:24.453109 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:24.453127 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:24.453169 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:29:24.453941 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:24.453961 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:24.454360 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:24.454733 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:24.455574 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.456594 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.457563 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.458061 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.458221 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.460978 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:24.460984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:24.461003 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:24.461038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:24.462925 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.463442 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.463447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.463452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.463580 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.463869 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.463871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.463872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.464347 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:24.464526 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:24.464917 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 04:29:24.465288 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":163,"threads_started":1}
I20260806 04:29:24.466801 31463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } } }
I20260806 04:29:24.466924 31463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:24.467137 31463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a6eb272ebff41fb8b0b8b63202b887d. Latest consensus state: current_term: 1 leader_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a6eb272ebff41fb8b0b8b63202b887d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46843 } } }
I20260806 04:29:24.467221 31463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:24.469166 31474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:24.472131 31474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:24.478487 31474 catalog_manager.cc:1347] Generated new cluster ID: 3c84bf4025bc41b0a483cd3d5d4e37bb
I20260806 04:29:24.478592 31474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:24.495237 31474 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:24.496315 31474 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:24.505530 31474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Generated new TSK 0
I20260806 04:29:24.506345 31474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:24.541477 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46843
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:24.664953 31491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:24.665275 31491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:24.665335 31491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:24.669808 31491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:24.669888 31491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:24.669932 31491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:29:24.669946 31491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:24.669991 31491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:24.675683 31491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:24.676978 31491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:24.678139 31491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:24.688424 31497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.688418 31500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.688418 31496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:24.688607 31491 server_base.cc:1034] running on GCE node
I20260806 04:29:24.689285 31491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:24.690032 31491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:24.691206 31491 hybrid_clock.cc:648] HybridClock initialized: now 1785990564691180 us; error 39 us; skew 500 ppm
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:24.694089 31491 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:24.695111 31491 webserver.cc:460] Webserver started at http://127.28.94.1:41991/ using document root <none> and password file <none>
I20260806 04:29:24.695720 31491 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:24.695793 31491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:24.696020 31491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:24.698251 31491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "c6dbcd51ecb74418a11b58fc93538cdb"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "1ee544383f3a76be72d904535398b5ea"
server_key_iv: "3f962e5c0dc3b00341dd1cbcf39b40f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.698915 31491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "c6dbcd51ecb74418a11b58fc93538cdb"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "1ee544383f3a76be72d904535398b5ea"
server_key_iv: "3f962e5c0dc3b00341dd1cbcf39b40f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.703358 31491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:24.706271 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:24.707633 31491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:24.707787 31491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "c6dbcd51ecb74418a11b58fc93538cdb"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "1ee544383f3a76be72d904535398b5ea"
server_key_iv: "3f962e5c0dc3b00341dd1cbcf39b40f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.707911 31491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:24.721650 31491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:24.724993 31491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:24.725206 31491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:24.726878 31491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:24.726938 31491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:24.726995 31491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:24.727025 31491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:24.738119 31491 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:34465
I20260806 04:29:24.738291 31674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:34465 every 8 connection(s)
I20260806 04:29:24.739148 31491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:24.740243 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 31491
I20260806 04:29:24.740355 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:24.740662 29048 external_mini_cluster.cc:1428] Setting key 34cf6e1215105c9458f32e7979b29fc0
I20260806 04:29:24.742350 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.727724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:44099 (local address 127.28.94.62:46843)
0806 04:29:24.727846 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:24.727850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:24.729084 (+ 1234us) server_negotiation.cc:407] Connection header received
0806 04:29:24.730230 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:24.730235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:24.730306 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:24.730368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:24.731999 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.732602 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.733896 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.734249 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.734379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.738379 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:24.738383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:24.738386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:24.738424 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:24.739940 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.740577 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.740582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.740584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.740691 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.741079 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.741082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.741084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.741312 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:24.741409 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:24.742068 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 04:29:24.742221 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:29:24.743566 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.728348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.1:44099)
0806 04:29:24.728886 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:24.728972 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:24.729941 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:24.730817 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:24.730824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:24.731195 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:24.731832 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.731850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.732745 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.732749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.733761 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.733774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.734461 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.734468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.734725 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.736150 (+ 1425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:24.736188 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:24.738115 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:24.740105 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.740112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.740129 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.740460 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.740779 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.740783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.740785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.740975 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.741412 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:24.741420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:24.741771 (+ 351us) client_negotiation.cc:769] Sending connection context
0806 04:29:24.741995 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:29:24.742417 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":140,"threads_started":1}
I20260806 04:29:24.745306 31675 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46843
I20260806 04:29:24.745471 31675 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:24.746119 31675 heartbeater.cc:511] Master 127.28.94.62:46843 requested a full tablet report, sending...
I20260806 04:29:24.748030 31394 ts_manager.cc:194] Registered new tserver with Master: c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465)
I20260806 04:29:24.749651 31394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:44099
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:24.754860 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.747739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33871 (local address 127.28.94.62:46843)
0806 04:29:24.747863 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:24.747868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:24.748116 (+ 248us) server_negotiation.cc:407] Connection header received
0806 04:29:24.748252 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:24.748255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:24.748298 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:24.748339 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:24.748935 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.749464 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.750406 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.750641 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.750732 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.751912 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:24.751916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:24.751918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:24.751942 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:24.753267 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.753681 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.753685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.753686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.753763 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.754014 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.754018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.754019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.754221 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:24.754310 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:24.754575 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 04:29:24.754692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:24.754892 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.747677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.1:33871)
0806 04:29:24.748036 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:24.748048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:24.748144 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:24.748465 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:24.748467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:24.748687 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:24.748797 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.748803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.749601 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.749604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.750254 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.750265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.750755 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.750759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.750853 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.751373 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:24.751386 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:24.751768 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:24.753375 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.753379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.753381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.753590 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.753845 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.753849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.753851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.753933 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.754312 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:24.754315 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:24.754411 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 04:29:24.754499 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:29:24.754658 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":5,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:24.772979 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46843
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:24.895741 31688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:24.896064 31688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:24.896137 31688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:24.900511 31688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:24.900609 31688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:24.900671 31688 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:29:24.900695 31688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:24.900754 31688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:24.906939 31688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:24.908213 31688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:24.909406 31688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:24.920028 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.920051 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:24.920048 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:24.920387 31688 server_base.cc:1034] running on GCE node
I20260806 04:29:24.920832 31688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:24.921615 31688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:24.922829 31688 hybrid_clock.cc:648] HybridClock initialized: now 1785990564922791 us; error 47 us; skew 500 ppm
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:24.926038 31688 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:24.927189 31688 webserver.cc:460] Webserver started at http://127.28.94.2:41773/ using document root <none> and password file <none>
I20260806 04:29:24.927839 31688 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:24.927919 31688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:24.928161 31688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:24.930631 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "b94d09cbc82c435f9b3c5ae610613d36"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "52788125ca8ffff6d234554646df8f7e"
server_key_iv: "604e58b001bc2561a6515fb8fc4791ee"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.931306 31688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "b94d09cbc82c435f9b3c5ae610613d36"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "52788125ca8ffff6d234554646df8f7e"
server_key_iv: "604e58b001bc2561a6515fb8fc4791ee"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.935633 31688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:29:24.938607 31710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:24.940094 31688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:24.940253 31688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "b94d09cbc82c435f9b3c5ae610613d36"
format_stamp: "Formatted at 2026-08-06 04:29:24 on dist-test-slave-ncp9"
server_key: "52788125ca8ffff6d234554646df8f7e"
server_key_iv: "604e58b001bc2561a6515fb8fc4791ee"
server_key_version: "encryptionkey@0"
I20260806 04:29:24.940382 31688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:24.953394 31688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:24.956799 31688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:24.957054 31688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:24.958734 31688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:24.958794 31688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:24.958853 31688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:24.958881 31688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:24 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:24.970042 31688 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42665
I20260806 04:29:24.970082 31862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42665 every 8 connection(s)
I20260806 04:29:24.971114 31688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:24.971335 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 31688
I20260806 04:29:24.971444 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:24.971761 29048 external_mini_cluster.cc:1428] Setting key 7852ab0fe0a5d5dcf81e7f6c6cf5a554
I20260806 04:29:24.974454 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.959700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:59867 (local address 127.28.94.62:46843)
0806 04:29:24.959862 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:24.959867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:24.960863 (+ 996us) server_negotiation.cc:407] Connection header received
0806 04:29:24.962171 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:24.962175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:24.962257 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:24.962336 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:24.963881 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.964533 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.966412 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.966743 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.966846 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.970201 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:24.970205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:24.970207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:24.970244 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:24.971888 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.972612 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.972616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.972618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.972722 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.973175 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.973179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.973181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.973460 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:24.973567 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:24.974173 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 04:29:24.974319 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:29:24.975622 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.960162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.2:59867)
0806 04:29:24.960667 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:24.960722 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:24.961916 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:24.962547 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:24.962554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:24.963017 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:24.963709 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.963728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.964680 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.964685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.966262 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.966280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.966913 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.966918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.967176 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.967982 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:24.968008 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:24.969961 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:24.972046 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.972052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.972071 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.972441 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.972822 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.972826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.972828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.973048 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.973583 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:24.973589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:24.973867 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 04:29:24.974093 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 04:29:24.974480 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260806 04:29:24.977003 31864 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46843
I20260806 04:29:24.977180 31864 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:24.977845 31864 heartbeater.cc:511] Master 127.28.94.62:46843 requested a full tablet report, sending...
I20260806 04:29:24.979144 31396 ts_manager.cc:194] Registered new tserver with Master: b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665)
I20260806 04:29:24.979985 31396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:59867
I20260806 04:29:24.987697 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.979737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.2:60741)
0806 04:29:24.980169 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:24.980187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:24.980431 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:24.980713 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:24.980715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:24.980968 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:24.981086 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.981092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.982062 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.982064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.982644 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:24.982651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.983186 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.983188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:24.983255 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.983782 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:24.983796 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:24.984155 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:24.986073 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.986076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.986078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.986290 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.986589 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:24.986591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:24.986593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:24.986663 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:24.987213 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:24.987215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:24.987308 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 04:29:24.987383 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:29:24.987531 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":100,"threads_started":1}
I20260806 04:29:24.987782 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:24.979830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:60741 (local address 127.28.94.62:46843)
0806 04:29:24.979986 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:24.979991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:24.980256 (+ 265us) server_negotiation.cc:407] Connection header received
0806 04:29:24.980533 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:24.980537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:24.980591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:24.980683 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:24.981201 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.981932 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.982760 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:24.983077 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:24.983182 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:24.984310 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:24.984315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:24.984318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:24.984349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:24.985940 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.986392 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.986397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.986399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.986504 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:24.986757 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:24.986761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:24.986763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:24.987122 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:24.987238 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:24.987468 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 04:29:24.987608 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:25.008154 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46843
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:25.135195 31875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:25.135551 31875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:25.135617 31875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:25.139757 31875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:25.139835 31875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:25.139876 31875 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:29:25.139889 31875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:25.139930 31875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:25.145651 31875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:25.146886 31875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:25.147969 31875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:25.158378 31881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:25.158378 31880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:25.158390 31883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:25.158703 31875 server_base.cc:1034] running on GCE node
I20260806 04:29:25.159157 31875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:25.159848 31875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:25.160959 31875 hybrid_clock.cc:648] HybridClock initialized: now 1785990565160937 us; error 35 us; skew 500 ppm
Aug 06 04:29:25 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990565, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:25.163789 31875 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:25.164755 31875 webserver.cc:460] Webserver started at http://127.28.94.3:42279/ using document root <none> and password file <none>
I20260806 04:29:25.165385 31875 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:25.165448 31875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:25.165645 31875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:25.167966 31875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
format_stamp: "Formatted at 2026-08-06 04:29:25 on dist-test-slave-ncp9"
server_key: "f40cc04ee174e14165439b8b8fea8093"
server_key_iv: "9a2c36099562e03c24b74399a1f0c465"
server_key_version: "encryptionkey@0"
I20260806 04:29:25.168550 31875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
format_stamp: "Formatted at 2026-08-06 04:29:25 on dist-test-slave-ncp9"
server_key: "f40cc04ee174e14165439b8b8fea8093"
server_key_iv: "9a2c36099562e03c24b74399a1f0c465"
server_key_version: "encryptionkey@0"
I20260806 04:29:25.172676 31875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:29:25.175587 31891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:25.177003 31875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:25.177137 31875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
format_stamp: "Formatted at 2026-08-06 04:29:25 on dist-test-slave-ncp9"
server_key: "f40cc04ee174e14165439b8b8fea8093"
server_key_iv: "9a2c36099562e03c24b74399a1f0c465"
server_key_version: "encryptionkey@0"
I20260806 04:29:25.177266 31875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:25.193076 31875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:25.196120 31875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:25.196290 31875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:25.197923 31875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:25.197979 31875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:25.198024 31875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:25.198051 31875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:25 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990565, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:25.208765 31875 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:37863
I20260806 04:29:25.208815 32044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:37863 every 8 connection(s)
I20260806 04:29:25.209728 31875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:25.212299 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.198733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:60017 (local address 127.28.94.62:46843)
0806 04:29:25.198856 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.198860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.199975 (+ 1115us) server_negotiation.cc:407] Connection header received
0806 04:29:25.201011 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.201016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.201078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.201135 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:25.202791 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.203548 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.205318 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.205577 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.205681 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.208860 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:25.208864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:25.208867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:25.208929 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:25.210460 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.210965 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.210968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.210969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.211060 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.211336 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.211338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.211339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.211515 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:25.211593 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.212031 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.212177 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:25.213220 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.199291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.3:60017)
0806 04:29:25.199800 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.199844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.200723 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.201553 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.201561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.201958 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:25.202642 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.202660 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.203673 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.203677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.205152 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.205177 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.205797 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.205802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.205998 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.206831 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:25.206852 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:25.208659 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:25.210579 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.210586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.210601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.210874 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.211137 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.211139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.211141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.211269 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.211590 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:25.211595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:25.211838 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.211989 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.212281 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
I20260806 04:29:25.214393 32045 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46843
I20260806 04:29:25.214545 32045 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:25.215044 32045 heartbeater.cc:511] Master 127.28.94.62:46843 requested a full tablet report, sending...
I20260806 04:29:25.216071 31394 ts_manager.cc:194] Registered new tserver with Master: 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:25.216711 31394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:60017
I20260806 04:29:25.217100 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 31875
I20260806 04:29:25.217217 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:25.217530 29048 external_mini_cluster.cc:1428] Setting key de26ea64cb5ecb6b4f69b1a1a5c0aab9
I20260806 04:29:25.219817 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:29:25.223994 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.216560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.28.94.3:58009)
0806 04:29:25.216993 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.217007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.217137 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.217422 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.217425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.217638 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:25.217753 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.217760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.218646 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.218650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.219434 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.219445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.219900 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.219904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.219995 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.220463 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:25.220477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:25.220917 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:25.222592 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.222594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.222596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.222779 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.223043 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.223047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.223049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.223126 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.223478 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:25.223481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:25.223574 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.223659 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.223807 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260806 04:29:25.223987 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.216669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:58009 (local address 127.28.94.62:46843)
0806 04:29:25.216789 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.216794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.217070 (+ 276us) server_negotiation.cc:407] Connection header received
0806 04:29:25.217264 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.217267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.217321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.217403 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:25.217917 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.218514 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.219536 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.219794 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.219872 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.221049 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:25.221052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:25.221055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:25.221078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:25.222484 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.222876 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.222879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.222881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.222962 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.223213 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.223216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.223218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.223391 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:25.223478 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.223730 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.223833 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Aug 06 04:29:25 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:29:25 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:25.247603 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.239866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.28.94.62:46843)
0806 04:29:25.239987 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.239991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.240004 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:29:25.240051 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.240054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.240097 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.240168 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:25.240934 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.241461 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.242257 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.242456 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.242523 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.244948 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:25.244952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:25.244955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:25.244984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:25.246207 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.246643 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.246646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.246647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.246737 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.246925 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.246927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.246928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.247098 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:25.247189 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.247352 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.247449 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:25.250599 31394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:25.253775 31394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:25.269260 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.263730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.28.94.2:42665)
0806 04:29:25.264072 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.264084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.264318 (+ 234us) server_negotiation.cc:407] Connection header received
0806 04:29:25.264387 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.264392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.264601 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.264755 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:25.265162 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.266335 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.267783 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268474 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268583 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.268776 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.268983 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.269084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260806 04:29:25.269248 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.263124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42665 (local address 127.0.0.1:46762)
0806 04:29:25.264134 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.264146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.264232 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.264931 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.264934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.264952 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:25.265043 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.265047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.266447 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.266451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.267682 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.267692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268591 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.268727 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.268772 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.268911 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.269078 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":6,"thread_start_us":91,"threads_started":1}
I20260806 04:29:25.269599 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.263640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.28.94.1:34465)
0806 04:29:25.264262 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.264269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.264292 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:29:25.264373 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.264379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.264602 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.264755 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:25.265142 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.266390 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.267769 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268577 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.268812 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.269364 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.269474 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":86,"threads_started":1}
I20260806 04:29:25.269789 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.262905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.28.94.3:37863)
0806 04:29:25.264341 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.264355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.264380 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:29:25.264473 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.264478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.264682 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.265075 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:25.265220 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.265946 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.267714 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268546 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268669 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.268831 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.269535 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.269638 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":95,"threads_started":1}
I20260806 04:29:25.269865 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.262890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.0.0.1:42802)
0806 04:29:25.263740 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.263781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.263922 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.264765 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.264772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.264796 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:25.265015 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.265024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.266507 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.266510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.267634 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.267644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268879 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.269074 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.269110 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.269187 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.269278 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":112,"spinlock_wait_cycles":14848,"thread_start_us":102,"threads_started":1}
I20260806 04:29:25.269435 32068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.262773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:37863 (local address 127.0.0.1:39684)
0806 04:29:25.263713 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.263770 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.263922 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.264841 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.264844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.264858 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:25.265128 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.265133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.266059 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.266062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.267576 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.267588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.268882 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.268885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.269064 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.269114 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.269181 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.269278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":215,"spinlock_wait_cycles":7296,"thread_start_us":100,"threads_started":1}
I20260806 04:29:25.272749 31777 tablet_service.cc:1467] Processing CreateTablet for tablet 73102087767d432087fbd907b2b3a415 (DEFAULT_TABLE table=test-table [id=04056280acad4fadb008e2898caf4aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:25.272647 31579 tablet_service.cc:1467] Processing CreateTablet for tablet 73102087767d432087fbd907b2b3a415 (DEFAULT_TABLE table=test-table [id=04056280acad4fadb008e2898caf4aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:25.273839 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73102087767d432087fbd907b2b3a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.273859 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73102087767d432087fbd907b2b3a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.275691 31955 tablet_service.cc:1467] Processing CreateTablet for tablet 73102087767d432087fbd907b2b3a415 (DEFAULT_TABLE table=test-table [id=04056280acad4fadb008e2898caf4aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:25.276947 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73102087767d432087fbd907b2b3a415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.281206 32074 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Bootstrap starting.
I20260806 04:29:25.283370 32075 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Bootstrap starting.
I20260806 04:29:25.283728 32074 tablet_bootstrap.cc:654] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.284524 32074 log.cc:824] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:25.286121 32075 tablet_bootstrap.cc:654] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.286969 32077 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Bootstrap starting.
I20260806 04:29:25.287101 32075 log.cc:824] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:25.287164 32074 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: No bootstrap required, opened a new log
I20260806 04:29:25.287384 32074 ts_tablet_manager.cc:1397] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:29:25.289351 32077 tablet_bootstrap.cc:654] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.289621 32075 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: No bootstrap required, opened a new log
I20260806 04:29:25.289842 32075 ts_tablet_manager.cc:1397] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:29:25.290283 32077 log.cc:824] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:25.291347 32074 raft_consensus.cc:348] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.291628 32074 raft_consensus.cc:374] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.291703 32074 raft_consensus.cc:729] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.292164 32074 consensus_queue.cc:260] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.292773 32077 tablet_bootstrap.cc:492] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: No bootstrap required, opened a new log
I20260806 04:29:25.293021 32077 ts_tablet_manager.cc:1397] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:29:25.293339 32074 ts_tablet_manager.cc:1428] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:25.293752 31675 heartbeater.cc:503] Master 127.28.94.62:46843 was elected leader, sending a full tablet report...
I20260806 04:29:25.293694 32075 raft_consensus.cc:348] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.293977 32075 raft_consensus.cc:374] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.294045 32075 raft_consensus.cc:729] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d09cbc82c435f9b3c5ae610613d36, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.294538 32075 consensus_queue.cc:260] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.295644 32075 ts_tablet_manager.cc:1428] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:29:25.296057 31864 heartbeater.cc:503] Master 127.28.94.62:46843 was elected leader, sending a full tablet report...
I20260806 04:29:25.296941 32077 raft_consensus.cc:348] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.297216 32077 raft_consensus.cc:374] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.297278 32077 raft_consensus.cc:729] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb2818b9cd4c6ea2eeee7d6717b6b1, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.297726 32077 consensus_queue.cc:260] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.298784 32077 ts_tablet_manager.cc:1428] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:29:25.299106 32045 heartbeater.cc:503] Master 127.28.94.62:46843 was elected leader, sending a full tablet report...
W20260806 04:29:25.460784 32046 tablet.cc:2369] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:25.473148 31865 tablet.cc:2369] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:25.490718 31677 tablet.cc:2369] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:25.578344 32082 raft_consensus.cc:482] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:25.578642 32082 raft_consensus.cc:504] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.579999 32082 leader_election.cc:288] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863), b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665)
I20260806 04:29:25.585793 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.580500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33265 (local address 127.28.94.3:37863)
0806 04:29:25.580716 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.580721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.580923 (+ 202us) server_negotiation.cc:407] Connection header received
0806 04:29:25.581029 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.581032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.581096 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.581177 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:25.581710 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.582565 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.584409 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.585195 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.585324 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.585386 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.585557 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.585642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 04:29:25.585793 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.580423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:60745 (local address 127.28.94.2:42665)
0806 04:29:25.580568 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.580584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.580843 (+ 259us) server_negotiation.cc:407] Connection header received
0806 04:29:25.581031 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.581033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.581090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.581177 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:25.581755 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.582565 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.584405 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.585107 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.585236 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.585322 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.585557 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.585647 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:29:25.585850 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.580363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:37863 (local address 127.28.94.1:33265)
0806 04:29:25.580795 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.580806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.580930 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.581237 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.581241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.581287 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:25.581574 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.581582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.582684 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.582688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.584217 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.584235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.585306 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.585310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.585407 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.585446 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.585556 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.585644 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":163,"spinlock_wait_cycles":15360,"thread_start_us":113,"threads_started":1}
I20260806 04:29:25.585850 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.580339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42665 (local address 127.28.94.1:60745)
0806 04:29:25.580734 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.580749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.580929 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.581237 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.581241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.581286 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:25.581574 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.581582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.582684 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.582688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.584218 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.584235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.585252 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.585257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.585389 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.585456 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.585529 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.585643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":260,"spinlock_wait_cycles":26624,"thread_start_us":183,"threads_started":1}
I20260806 04:29:25.586637 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73102087767d432087fbd907b2b3a415" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" is_pre_election: true
I20260806 04:29:25.587030 31983 raft_consensus.cc:2454] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:25.586980 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73102087767d432087fbd907b2b3a415" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:25.587289 31806 raft_consensus.cc:2454] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:25.587622 31516 leader_election.cc:302] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb2818b9cd4c6ea2eeee7d6717b6b1, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:25.587970 32082 raft_consensus.cc:2781] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:25.588063 32082 raft_consensus.cc:482] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:25.588095 32082 raft_consensus.cc:3037] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.589241 32082 raft_consensus.cc:504] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.589619 32082 leader_election.cc:288] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Requested vote from peers 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863), b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665)
I20260806 04:29:25.590124 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73102087767d432087fbd907b2b3a415" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36"
I20260806 04:29:25.590116 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73102087767d432087fbd907b2b3a415" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
I20260806 04:29:25.590269 31806 raft_consensus.cc:3037] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.590268 31983 raft_consensus.cc:3037] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.591647 31806 raft_consensus.cc:2454] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:25.592067 31514 leader_election.cc:302] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:25.592131 31983 raft_consensus.cc:2454] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:25.592288 32082 raft_consensus.cc:2781] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:25.592540 32082 raft_consensus.cc:686] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Becoming Leader. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Running, Role: LEADER
I20260806 04:29:25.592934 32082 consensus_queue.cc:237] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } }
I20260806 04:29:25.596583 31396 catalog_manager.cc:5495] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:29:25 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990564, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:25.655107 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.646906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46843 (local address 127.0.0.1:54520)
0806 04:29:25.647216 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.647228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.647349 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.647581 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.647584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.648035 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:25.648150 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.648157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.648930 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.648934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.649491 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.649499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.649918 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.649921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.650000 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.650578 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:25.650591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:25.652044 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:25.653462 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.653476 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.653484 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.653745 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.654006 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:25.654010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:25.654012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:25.654091 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:25.654479 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:25.654487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:25.654609 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.654720 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.654906 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":128,"threads_started":1}
I20260806 04:29:25.655107 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.646963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.28.94.62:46843)
0806 04:29:25.647086 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.647091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.647302 (+ 211us) server_negotiation.cc:407] Connection header received
0806 04:29:25.647446 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.647449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.647486 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.647547 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:25.648276 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.648789 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.649603 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.649812 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.649899 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.652201 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:25.652204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:25.652206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:25.652231 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:25.653355 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.653841 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.653844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.653845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.653922 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:25.654169 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:25.654172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:25.654174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:25.654371 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:25.654467 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.654783 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.654983 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":40}
I20260806 04:29:25.659651 31396 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:54520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:29:25.660449 31396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:25.666025 31777 tablet_service.cc:1467] Processing CreateTablet for tablet e48d8883f961497f9bfe56e7c894ad0d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71b5fdb6f8264821bf8594f2c6db0637]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:25.665832 31579 tablet_service.cc:1467] Processing CreateTablet for tablet e48d8883f961497f9bfe56e7c894ad0d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71b5fdb6f8264821bf8594f2c6db0637]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:25.665948 31955 tablet_service.cc:1467] Processing CreateTablet for tablet e48d8883f961497f9bfe56e7c894ad0d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71b5fdb6f8264821bf8594f2c6db0637]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:25.666739 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48d8883f961497f9bfe56e7c894ad0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.666739 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48d8883f961497f9bfe56e7c894ad0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.666762 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48d8883f961497f9bfe56e7c894ad0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:25.669945 32077 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Bootstrap starting.
I20260806 04:29:25.669960 32075 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: Bootstrap starting.
I20260806 04:29:25.670936 32077 tablet_bootstrap.cc:654] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.670986 32075 tablet_bootstrap.cc:654] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.671615 32074 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: Bootstrap starting.
I20260806 04:29:25.672560 32075 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: No bootstrap required, opened a new log
I20260806 04:29:25.672672 32075 ts_tablet_manager.cc:1397] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:25.672833 32074 tablet_bootstrap.cc:654] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:25.673174 32075 raft_consensus.cc:348] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.673286 32075 raft_consensus.cc:374] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.673316 32075 raft_consensus.cc:729] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d09cbc82c435f9b3c5ae610613d36, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.673460 32075 consensus_queue.cc:260] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.673779 32075 ts_tablet_manager.cc:1428] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:25.673804 32083 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.673967 32083 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.674579 32074 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: No bootstrap required, opened a new log
I20260806 04:29:25.674674 32074 ts_tablet_manager.cc:1397] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:25.674881 32077 tablet_bootstrap.cc:492] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: No bootstrap required, opened a new log
I20260806 04:29:25.674984 32077 ts_tablet_manager.cc:1397] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 04:29:25.675230 32074 raft_consensus.cc:348] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.675338 32074 raft_consensus.cc:374] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.675374 32074 raft_consensus.cc:729] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.675454 32077 raft_consensus.cc:348] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.675503 32074 consensus_queue.cc:260] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.675588 32077 raft_consensus.cc:374] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:25.675631 32077 raft_consensus.cc:729] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb2818b9cd4c6ea2eeee7d6717b6b1, State: Initialized, Role: FOLLOWER
I20260806 04:29:25.675770 32077 consensus_queue.cc:260] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.676047 32077 ts_tablet_manager.cc:1428] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:25.676124 32084 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.676276 32084 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.676606 32074 ts_tablet_manager.cc:1428] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:29:25.676591 32082 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.676746 32082 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.876439 32082 raft_consensus.cc:482] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:25.876636 32082 raft_consensus.cc:504] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.877221 32082 leader_election.cc:288] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:25.877758 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48d8883f961497f9bfe56e7c894ad0d" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" is_pre_election: true
I20260806 04:29:25.877761 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48d8883f961497f9bfe56e7c894ad0d" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:25.877921 31983 raft_consensus.cc:2454] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:25.877950 31806 raft_consensus.cc:2454] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:25.878257 31516 leader_election.cc:302] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb2818b9cd4c6ea2eeee7d6717b6b1, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:25.878465 32082 raft_consensus.cc:2781] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:25.878533 32082 raft_consensus.cc:482] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:25.878575 32082 raft_consensus.cc:3037] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.879595 32082 raft_consensus.cc:504] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.879972 32082 leader_election.cc:288] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Requested vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:25.880417 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48d8883f961497f9bfe56e7c894ad0d" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
I20260806 04:29:25.880412 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48d8883f961497f9bfe56e7c894ad0d" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36"
I20260806 04:29:25.880546 31983 raft_consensus.cc:3037] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.880553 31806 raft_consensus.cc:3037] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:25.881640 31806 raft_consensus.cc:2454] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:25.881664 31983 raft_consensus.cc:2454] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:25.882014 31514 leader_election.cc:302] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:25.882231 32082 raft_consensus.cc:2781] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:25.882344 32082 raft_consensus.cc:686] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Becoming Leader. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Running, Role: LEADER
I20260806 04:29:25.882503 32082 consensus_queue.cc:237] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:25.883224 32088 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: TxnStatusTablet state changed. Reason: New leader c6dbcd51ecb74418a11b58fc93538cdb. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.883349 32088 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:25.883615 32100 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:29:25.883853 32100 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:29:25.884441 31396 catalog_manager.cc:5495] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:25.914054 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.909455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.0.0.1:42810)
0806 04:29:25.909745 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.909761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.909881 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.910239 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.910243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.910251 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:25.910376 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.910383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.911147 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.911150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.912186 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.912200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.912711 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.912715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.912818 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.912928 (+ 110us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:25.913694 (+ 766us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:25.913709 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.913783 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.913894 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260806 04:29:25.914258 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.909545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.28.94.1:34465)
0806 04:29:25.909855 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.909862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.909877 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:25.910030 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.910033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.910093 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.910202 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:25.910485 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.911048 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.912302 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.912591 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.912677 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.913095 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:25.913598 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:25.913698 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.913793 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.913852 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":129,"threads_started":1}
I20260806 04:29:25.920284 31983 raft_consensus.cc:1261] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:25.920284 31806 raft_consensus.cc:1261] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:25.921236 32082 consensus_queue.cc:1035] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:25.921440 32088 consensus_queue.cc:1035] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:25.925948 32084 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: TxnStatusTablet state changed. Reason: New leader c6dbcd51ecb74418a11b58fc93538cdb. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.926131 32084 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.926280 32083 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: TxnStatusTablet state changed. Reason: New leader c6dbcd51ecb74418a11b58fc93538cdb. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.926411 32083 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.932055 32082 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.932246 32082 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:25.932391 32082 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.932471 32082 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P c6dbcd51ecb74418a11b58fc93538cdb: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:25.932471 32084 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.932574 32084 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P 0efb2818b9cd4c6ea2eeee7d6717b6b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.932554 32105 mvcc.cc:205] Tried to move back new op lower bound from 7315417358001090560 to 7315417357856268288. Current Snapshot: MvccSnapshot[applied={T|T < 7315417358001090560}]
I20260806 04:29:25.932619 32083 tablet_replica.cc:439] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } } }
I20260806 04:29:25.932760 32083 tablet_replica.cc:442] T e48d8883f961497f9bfe56e7c894ad0d P b94d09cbc82c435f9b3c5ae610613d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:25.933925 32106 mvcc.cc:205] Tried to move back new op lower bound from 7315417358001090560 to 7315417357856268288. Current Snapshot: MvccSnapshot[applied={T|T < 7315417358001090560}]
I20260806 04:29:25.937147 32108 mvcc.cc:205] Tried to move back new op lower bound from 7315417358001090560 to 7315417357856268288. Current Snapshot: MvccSnapshot[applied={T|T < 7315417358001090560}]
I20260806 04:29:25.944080 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.940623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.28.94.1:34465)
0806 04:29:25.940774 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.940779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.941065 (+ 286us) server_negotiation.cc:407] Connection header received
0806 04:29:25.941220 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.941223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.941277 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.941338 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:25.941604 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.942117 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.943011 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.943245 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.943339 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.943618 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:25.943712 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:25.943771 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.943868 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.943928 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:29:25.957142 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.953315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:40845 (local address 127.28.94.1:34465)
0806 04:29:25.953449 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:25.953454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:25.953662 (+ 208us) server_negotiation.cc:407] Connection header received
0806 04:29:25.953799 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:25.953802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:25.953847 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:25.953907 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:25.954201 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.954720 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.955992 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.956218 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.956293 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.956680 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:25.956761 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:25.956807 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:25.956949 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 04:29:25.957007 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:25.957213 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:25.953251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.28.94.1:40845)
0806 04:29:25.953574 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:25.953589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:25.953714 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:25.953937 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:25.953940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:25.953947 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:25.954081 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.954088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.954824 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.954828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.955855 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:25.955866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:25.956328 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:25.956332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:25.956438 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:25.956509 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:25.956836 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:25.956843 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:29:25.956930 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:29:25.957026 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260806 04:29:25.965898 31983 raft_consensus.cc:1261] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:25.966053 31806 raft_consensus.cc:1261] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:25.966476 32082 consensus_queue.cc:1035] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:25.966665 32124 consensus_queue.cc:1035] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:26.027923 31396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:26.028124 31396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:26.031124 31396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Sending DeleteTablet for 3 replicas of tablet 73102087767d432087fbd907b2b3a415
I20260806 04:29:26.032135 31955 tablet_service.cc:1514] Processing DeleteTablet for tablet 73102087767d432087fbd907b2b3a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:26 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:39684
I20260806 04:29:26.032353 31777 tablet_service.cc:1514] Processing DeleteTablet for tablet 73102087767d432087fbd907b2b3a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:26 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260806 04:29:26.032553 31578 tablet_service.cc:1514] Processing DeleteTablet for tablet 73102087767d432087fbd907b2b3a415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:26 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:42802
I20260806 04:29:26.032775 32131 tablet_replica.cc:330] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: stopping tablet replica
I20260806 04:29:26.033190 32131 raft_consensus.cc:2229] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:26.033223 32134 tablet_replica.cc:330] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: stopping tablet replica
I20260806 04:29:26.033468 32131 raft_consensus.cc:2258] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:26.033651 32134 raft_consensus.cc:2229] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:26.033980 31393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:29:26.034349 32134 raft_consensus.cc:2258] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 04:29:26.034454 31393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:26.035084 32131 ts_tablet_manager.cc:1905] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:26.035990 32132 tablet_replica.cc:330] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: stopping tablet replica
I20260806 04:29:26.036288 32132 raft_consensus.cc:2229] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:26.036482 32132 raft_consensus.cc:2258] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:26.037395 32134 ts_tablet_manager.cc:1905] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:26.039008 32131 ts_tablet_manager.cc:1918] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:26.039127 32131 log.cc:1196] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/73102087767d432087fbd907b2b3a415
I20260806 04:29:26.039453 32131 ts_tablet_manager.cc:1939] T 73102087767d432087fbd907b2b3a415 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting consensus metadata
I20260806 04:29:26.040886 31578 tablet_service.cc:1467] Processing CreateTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 (DEFAULT_TABLE table=test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:26.041122 32134 ts_tablet_manager.cc:1918] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:26.041309 32134 log.cc:1196] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/73102087767d432087fbd907b2b3a415
I20260806 04:29:26.041200 31578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a9d6ce5a194c569fb65aaf8677c161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:26.041357 31777 tablet_service.cc:1467] Processing CreateTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 (DEFAULT_TABLE table=test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:26.041661 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a9d6ce5a194c569fb65aaf8677c161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:26.041857 32134 ts_tablet_manager.cc:1939] T 73102087767d432087fbd907b2b3a415 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting consensus metadata
I20260806 04:29:26.043397 31375 catalog_manager.cc:4841] TS 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863): tablet 73102087767d432087fbd907b2b3a415 (table test-table [id=04056280acad4fadb008e2898caf4aa7]) successfully deleted
I20260806 04:29:26.043803 32074 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Bootstrap starting.
I20260806 04:29:26.044377 31955 tablet_service.cc:1467] Processing CreateTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 (DEFAULT_TABLE table=test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:26.044704 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a9d6ce5a194c569fb65aaf8677c161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:26.044792 32074 tablet_bootstrap.cc:654] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:26.045079 32075 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Bootstrap starting.
I20260806 04:29:26.045184 32132 ts_tablet_manager.cc:1905] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:26.045950 32075 tablet_bootstrap.cc:654] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:26.047215 32077 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Bootstrap starting.
I20260806 04:29:26.047473 31378 catalog_manager.cc:4841] TS c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465): tablet 73102087767d432087fbd907b2b3a415 (table test-table [id=04056280acad4fadb008e2898caf4aa7]) successfully deleted
I20260806 04:29:26.048157 32077 tablet_bootstrap.cc:654] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:26.048647 32132 ts_tablet_manager.cc:1918] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:26.048801 32132 log.cc:1196] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/73102087767d432087fbd907b2b3a415
I20260806 04:29:26.049229 32132 ts_tablet_manager.cc:1939] T 73102087767d432087fbd907b2b3a415 P b94d09cbc82c435f9b3c5ae610613d36: Deleting consensus metadata
I20260806 04:29:26.049484 32075 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: No bootstrap required, opened a new log
I20260806 04:29:26.049579 32075 ts_tablet_manager.cc:1397] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260806 04:29:26.049819 32077 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: No bootstrap required, opened a new log
I20260806 04:29:26.049906 32077 ts_tablet_manager.cc:1397] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:26.050102 32075 raft_consensus.cc:348] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.050231 32075 raft_consensus.cc:374] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:26.050263 32075 raft_consensus.cc:729] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d09cbc82c435f9b3c5ae610613d36, State: Initialized, Role: FOLLOWER
I20260806 04:29:26.050334 32077 raft_consensus.cc:348] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.050395 32075 consensus_queue.cc:260] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.050450 32077 raft_consensus.cc:374] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:26.050482 32074 tablet_bootstrap.cc:492] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: No bootstrap required, opened a new log
I20260806 04:29:26.050485 32077 raft_consensus.cc:729] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb2818b9cd4c6ea2eeee7d6717b6b1, State: Initialized, Role: FOLLOWER
I20260806 04:29:26.050575 32074 ts_tablet_manager.cc:1397] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 04:29:26.050673 32077 consensus_queue.cc:260] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.050956 32077 ts_tablet_manager.cc:1428] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:26.051014 32074 raft_consensus.cc:348] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.051160 32074 raft_consensus.cc:374] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:26.051196 32074 raft_consensus.cc:729] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Initialized, Role: FOLLOWER
I20260806 04:29:26.051100 32075 ts_tablet_manager.cc:1428] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:29:26.051324 32074 consensus_queue.cc:260] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.051584 32074 ts_tablet_manager.cc:1428] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:26.052830 31371 catalog_manager.cc:4841] TS b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665): tablet 73102087767d432087fbd907b2b3a415 (table test-table [id=04056280acad4fadb008e2898caf4aa7]) successfully deleted
I20260806 04:29:26.345155 32084 raft_consensus.cc:482] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:26.345329 32084 raft_consensus.cc:504] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.346527 32084 leader_election.cc:288] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465), b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665)
I20260806 04:29:26.347579 32124 raft_consensus.cc:482] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:26.347719 32124 raft_consensus.cc:504] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.348148 32124 leader_election.cc:288] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:26.348564 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" is_pre_election: true
I20260806 04:29:26.348582 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:26.348723 31983 raft_consensus.cc:2454] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:26.348723 31806 raft_consensus.cc:2454] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:26.349017 31514 leader_election.cc:302] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:26.349298 32124 raft_consensus.cc:2781] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:26.349359 32124 raft_consensus.cc:482] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:26.349400 32124 raft_consensus.cc:3037] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:26.350502 32124 raft_consensus.cc:504] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.350884 32124 leader_election.cc:288] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Requested vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:26.351384 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36"
I20260806 04:29:26.351409 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
I20260806 04:29:26.351531 31806 raft_consensus.cc:3037] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:26.351537 31983 raft_consensus.cc:3037] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:26.351835 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:26.346820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42665 (local address 127.28.94.3:37431)
0806 04:29:26.347106 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:26.347121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:26.347251 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:26.347518 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:26.347521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:26.347537 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:26.347655 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:26.347662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.348747 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.348751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:26.350277 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:26.350290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.351293 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.351296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:26.351449 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:26.351485 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:29:26.351562 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:29:26.351652 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":211,"thread_start_us":93,"threads_started":1}
I20260806 04:29:26.351858 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:26.347137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:57785 (local address 127.28.94.1:34465)
0806 04:29:26.347258 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:26.347262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:26.347277 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:26.347407 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:26.347410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:26.347450 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:26.347515 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:26.347871 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.348609 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.350435 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.350892 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.351088 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:26.351174 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:26.351640 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 04:29:26.351743 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:26.352211 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:26.346895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37431 (local address 127.28.94.2:42665)
0806 04:29:26.347215 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:26.347219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:26.347235 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:29:26.347365 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:26.347368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:26.347416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:26.347498 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:26.347786 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.348620 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.350433 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.351057 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.351170 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:26.351235 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:26.351983 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 04:29:26.352078 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20260806 04:29:26.352216 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:26.346822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.28.94.3:57785)
0806 04:29:26.347208 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:26.347223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:26.347317 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:26.347638 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:26.347641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:26.347660 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:26.347769 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:26.347776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.348747 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.348751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:26.350277 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:26.350290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:26.351279 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:26.351283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:26.351454 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:26.351487 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:29:26.351989 (+ 502us) client_negotiation.cc:240] Negotiation successful
0806 04:29:26.352075 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":212,"thread_start_us":66,"threads_started":1}
I20260806 04:29:26.352695 31983 raft_consensus.cc:2454] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:26.352702 31806 raft_consensus.cc:2454] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:26.353024 31514 leader_election.cc:302] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:26.353310 32124 raft_consensus.cc:2781] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:26.353396 32124 raft_consensus.cc:686] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Becoming Leader. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Running, Role: LEADER
I20260806 04:29:26.353468 31610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" is_pre_election: true
I20260806 04:29:26.353533 32124 consensus_queue.cc:237] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:26.353590 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a9d6ce5a194c569fb65aaf8677c161" candidate_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:26.353741 31806 raft_consensus.cc:2379] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0efb2818b9cd4c6ea2eeee7d6717b6b1 in current term 1: Already voted for candidate c6dbcd51ecb74418a11b58fc93538cdb in this term.
I20260806 04:29:26.354312 31895 leader_election.cc:302] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0efb2818b9cd4c6ea2eeee7d6717b6b1; no voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb
I20260806 04:29:26.354617 32084 raft_consensus.cc:2726] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:29:26.355434 31396 catalog_manager.cc:5495] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:26.432101 31983 raft_consensus.cc:1261] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:26.432124 31805 raft_consensus.cc:1261] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:26.432588 32124 consensus_queue.cc:1035] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:26.432811 32112 consensus_queue.cc:1035] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:29:26.457005 32105 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:29:28.023475 31394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:28.023667 31394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:28.026575 31394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Sending DeleteTablet for 3 replicas of tablet c5a9d6ce5a194c569fb65aaf8677c161
I20260806 04:29:28.027334 31579 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:42802
I20260806 04:29:28.027449 31955 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:39684
I20260806 04:29:28.027478 31777 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a9d6ce5a194c569fb65aaf8677c161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260806 04:29:28.027863 32177 tablet_replica.cc:330] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: stopping tablet replica
I20260806 04:29:28.027863 32174 tablet_replica.cc:330] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: stopping tablet replica
I20260806 04:29:28.028023 32174 raft_consensus.cc:2229] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:28.028069 32177 raft_consensus.cc:2229] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:28.028266 32177 raft_consensus.cc:2258] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.028321 32174 raft_consensus.cc:2258] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.029078 32177 ts_tablet_manager.cc:1905] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:28.029182 32174 ts_tablet_manager.cc:1905] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:28.031383 32176 tablet_replica.cc:330] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: stopping tablet replica
I20260806 04:29:28.031595 32176 raft_consensus.cc:2229] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:28.031774 32176 raft_consensus.cc:2258] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.032400 32174 ts_tablet_manager.cc:1918] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:29:28.032559 32174 log.cc:1196] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/c5a9d6ce5a194c569fb65aaf8677c161
I20260806 04:29:28.032536 32176 ts_tablet_manager.cc:1905] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:28.032657 32177 ts_tablet_manager.cc:1918] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:29:28.032824 32177 log.cc:1196] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/c5a9d6ce5a194c569fb65aaf8677c161
I20260806 04:29:28.032992 32174 ts_tablet_manager.cc:1939] T c5a9d6ce5a194c569fb65aaf8677c161 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting consensus metadata
I20260806 04:29:28.033159 32177 ts_tablet_manager.cc:1939] T c5a9d6ce5a194c569fb65aaf8677c161 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting consensus metadata
I20260806 04:29:28.033926 31375 catalog_manager.cc:4841] TS 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863): tablet c5a9d6ce5a194c569fb65aaf8677c161 (table test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]) successfully deleted
I20260806 04:29:28.033923 31378 catalog_manager.cc:4841] TS c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465): tablet c5a9d6ce5a194c569fb65aaf8677c161 (table test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]) successfully deleted
I20260806 04:29:28.035349 32176 ts_tablet_manager.cc:1918] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:29:28.035439 32176 log.cc:1196] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/c5a9d6ce5a194c569fb65aaf8677c161
I20260806 04:29:28.035658 32176 ts_tablet_manager.cc:1939] T c5a9d6ce5a194c569fb65aaf8677c161 P b94d09cbc82c435f9b3c5ae610613d36: Deleting consensus metadata
I20260806 04:29:28.036643 31371 catalog_manager.cc:4841] TS b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665): tablet c5a9d6ce5a194c569fb65aaf8677c161 (table test-table [id=f7bcbda1b5bd4292afef8263f3d6e7aa]) successfully deleted
Aug 06 04:29:28 dist-test-slave-ncp9 krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:29:28 dist-test-slave-ncp9 krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:28.060559 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:28.051800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.28.94.62:46843)
0806 04:29:28.052119 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:28.052129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:28.052146 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:29:28.052296 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:28.052300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:28.052350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:28.052429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:28.053244 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:28.053823 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:28.054695 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:28.054934 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:28.055029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:28.057509 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:28.057512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:28.057514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:28.057546 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:28.058854 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:28.059281 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:28.059285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:28.059287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:28.059372 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:28.059639 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:28.059643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:28.059645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:28.059862 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:28.059989 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:28.060238 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:29:28.060377 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":128,"threads_started":1}
I20260806 04:29:28.063305 31394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:28.063759 31394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:28.068611 31579 tablet_service.cc:1467] Processing CreateTablet for tablet f47914b2aa6743e2b024d174b6d973a9 (DEFAULT_TABLE table=test-table [id=3b41777667ea4250932f65190c9e028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.068686 31777 tablet_service.cc:1467] Processing CreateTablet for tablet f47914b2aa6743e2b024d174b6d973a9 (DEFAULT_TABLE table=test-table [id=3b41777667ea4250932f65190c9e028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.068749 31955 tablet_service.cc:1467] Processing CreateTablet for tablet f47914b2aa6743e2b024d174b6d973a9 (DEFAULT_TABLE table=test-table [id=3b41777667ea4250932f65190c9e028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.068997 31579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47914b2aa6743e2b024d174b6d973a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.068998 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47914b2aa6743e2b024d174b6d973a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.069079 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47914b2aa6743e2b024d174b6d973a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.071137 32186 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Bootstrap starting.
I20260806 04:29:28.071671 32188 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Bootstrap starting.
I20260806 04:29:28.072077 32186 tablet_bootstrap.cc:654] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.072543 32188 tablet_bootstrap.cc:654] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.073110 32187 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Bootstrap starting.
I20260806 04:29:28.073413 32186 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: No bootstrap required, opened a new log
I20260806 04:29:28.073567 32186 ts_tablet_manager.cc:1397] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:29:28.073632 32188 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: No bootstrap required, opened a new log
I20260806 04:29:28.073717 32188 ts_tablet_manager.cc:1397] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 04:29:28.074093 32188 raft_consensus.cc:348] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.074101 32186 raft_consensus.cc:348] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.074222 32186 raft_consensus.cc:374] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.074287 32186 raft_consensus.cc:729] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb2818b9cd4c6ea2eeee7d6717b6b1, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.074316 32188 raft_consensus.cc:374] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.074388 32188 raft_consensus.cc:729] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.074405 32187 tablet_bootstrap.cc:654] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.074584 32188 consensus_queue.cc:260] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.074577 32186 consensus_queue.cc:260] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.075001 32188 ts_tablet_manager.cc:1428] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:28.075435 32186 ts_tablet_manager.cc:1428] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 04:29:28.076356 32187 tablet_bootstrap.cc:492] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: No bootstrap required, opened a new log
I20260806 04:29:28.076458 32187 ts_tablet_manager.cc:1397] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:28.077006 32187 raft_consensus.cc:348] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.077111 32187 raft_consensus.cc:374] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.077196 32187 raft_consensus.cc:729] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d09cbc82c435f9b3c5ae610613d36, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.077337 32187 consensus_queue.cc:260] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.077745 32187 ts_tablet_manager.cc:1428] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:29:28.322253 32124 raft_consensus.cc:482] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:28.322413 32124 raft_consensus.cc:504] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.322887 32124 leader_election.cc:288] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:28.323390 31805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47914b2aa6743e2b024d174b6d973a9" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:28.323397 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47914b2aa6743e2b024d174b6d973a9" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" is_pre_election: true
I20260806 04:29:28.323540 31983 raft_consensus.cc:2454] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:28.323562 31805 raft_consensus.cc:2454] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:28.323904 31514 leader_election.cc:302] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:28.324111 32124 raft_consensus.cc:2781] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:28.324190 32124 raft_consensus.cc:482] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:28.324239 32124 raft_consensus.cc:3037] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.325347 32124 raft_consensus.cc:504] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.325711 32124 leader_election.cc:288] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Requested vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:28.326131 31805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47914b2aa6743e2b024d174b6d973a9" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36"
I20260806 04:29:28.326128 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47914b2aa6743e2b024d174b6d973a9" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
I20260806 04:29:28.326247 31983 raft_consensus.cc:3037] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.326256 31805 raft_consensus.cc:3037] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.327196 31805 raft_consensus.cc:2454] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:28.327210 31983 raft_consensus.cc:2454] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:28.327497 31514 leader_election.cc:302] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:28.327670 32124 raft_consensus.cc:2781] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:28.327764 32124 raft_consensus.cc:686] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Becoming Leader. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Running, Role: LEADER
I20260806 04:29:28.327908 32124 consensus_queue.cc:237] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.329799 31394 catalog_manager.cc:5495] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:28.378410 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:28.375145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.28.94.1:34465)
0806 04:29:28.375440 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:28.375448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:28.375459 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:29:28.375499 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:28.375501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:28.375544 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:28.375624 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:28.375887 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:28.376426 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:28.377354 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:28.377583 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:28.377673 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:28.377934 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:28.378017 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:28.378084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:28.378187 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 04:29:28.378244 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
I20260806 04:29:28.385532 31805 raft_consensus.cc:1261] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:28.385532 31983 raft_consensus.cc:1261] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:28.385926 32151 consensus_queue.cc:1035] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:28.386098 32163 consensus_queue.cc:1035] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:28.420246 31394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:28.420419 31394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:28.422675 31394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Sending DeleteTablet for 3 replicas of tablet f47914b2aa6743e2b024d174b6d973a9
I20260806 04:29:28.423332 31578 tablet_service.cc:1514] Processing DeleteTablet for tablet f47914b2aa6743e2b024d174b6d973a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:42802
I20260806 04:29:28.423408 31777 tablet_service.cc:1514] Processing DeleteTablet for tablet f47914b2aa6743e2b024d174b6d973a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260806 04:29:28.423391 31955 tablet_service.cc:1514] Processing DeleteTablet for tablet f47914b2aa6743e2b024d174b6d973a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:39684
I20260806 04:29:28.423611 32176 tablet_replica.cc:330] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: stopping tablet replica
I20260806 04:29:28.423611 32177 tablet_replica.cc:330] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: stopping tablet replica
I20260806 04:29:28.423717 32176 raft_consensus.cc:2229] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:28.423740 32177 raft_consensus.cc:2229] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:28.423831 32176 raft_consensus.cc:2258] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.423872 32177 raft_consensus.cc:2258] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.423887 32174 tablet_replica.cc:330] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: stopping tablet replica
I20260806 04:29:28.424002 32174 raft_consensus.cc:2229] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:28.424222 32174 raft_consensus.cc:2258] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:28.424321 31394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:29:28.424866 32177 ts_tablet_manager.cc:1905] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:28.424858 32176 ts_tablet_manager.cc:1905] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:29:28.424984 31394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:28.425099 32174 ts_tablet_manager.cc:1905] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:28.427712 32177 ts_tablet_manager.cc:1918] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:28.427712 32174 ts_tablet_manager.cc:1918] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:28.427714 32176 ts_tablet_manager.cc:1918] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:28.427807 32174 log.cc:1196] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/f47914b2aa6743e2b024d174b6d973a9
I20260806 04:29:28.427809 32176 log.cc:1196] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/f47914b2aa6743e2b024d174b6d973a9
I20260806 04:29:28.427927 32177 log.cc:1196] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/f47914b2aa6743e2b024d174b6d973a9
I20260806 04:29:28.428086 32176 ts_tablet_manager.cc:1939] T f47914b2aa6743e2b024d174b6d973a9 P b94d09cbc82c435f9b3c5ae610613d36: Deleting consensus metadata
I20260806 04:29:28.428279 32177 ts_tablet_manager.cc:1939] T f47914b2aa6743e2b024d174b6d973a9 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting consensus metadata
I20260806 04:29:28.428954 31371 catalog_manager.cc:4841] TS b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665): tablet f47914b2aa6743e2b024d174b6d973a9 (table test-table [id=3b41777667ea4250932f65190c9e028c]) successfully deleted
I20260806 04:29:28.429083 31375 catalog_manager.cc:4841] TS 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863): tablet f47914b2aa6743e2b024d174b6d973a9 (table test-table [id=3b41777667ea4250932f65190c9e028c]) successfully deleted
I20260806 04:29:28.429651 32174 ts_tablet_manager.cc:1939] T f47914b2aa6743e2b024d174b6d973a9 P c6dbcd51ecb74418a11b58fc93538cdb: Deleting consensus metadata
I20260806 04:29:28.430405 31378 catalog_manager.cc:4841] TS c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465): tablet f47914b2aa6743e2b024d174b6d973a9 (table test-table [id=3b41777667ea4250932f65190c9e028c]) successfully deleted
I20260806 04:29:28.431277 31578 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 (DEFAULT_TABLE table=test-table [id=66171c8014aa424fa1544ae3a9a68851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.431293 31777 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 (DEFAULT_TABLE table=test-table [id=66171c8014aa424fa1544ae3a9a68851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.431344 31955 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 (DEFAULT_TABLE table=test-table [id=66171c8014aa424fa1544ae3a9a68851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:28.431578 31578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b87e4fed24487afa5cf75b0419f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.431587 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b87e4fed24487afa5cf75b0419f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.431630 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b87e4fed24487afa5cf75b0419f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:28.433908 32186 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Bootstrap starting.
I20260806 04:29:28.434149 32188 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: Bootstrap starting.
I20260806 04:29:28.434886 32186 tablet_bootstrap.cc:654] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.434933 32188 tablet_bootstrap.cc:654] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.435560 32187 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: Bootstrap starting.
I20260806 04:29:28.436172 32188 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: No bootstrap required, opened a new log
I20260806 04:29:28.436266 32188 ts_tablet_manager.cc:1397] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 04:29:28.436479 32187 tablet_bootstrap.cc:654] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:28.436491 32186 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: No bootstrap required, opened a new log
I20260806 04:29:28.436582 32186 ts_tablet_manager.cc:1397] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:28.436725 32188 raft_consensus.cc:348] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.436838 32188 raft_consensus.cc:374] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.436877 32188 raft_consensus.cc:729] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.437039 32188 consensus_queue.cc:260] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.437100 32186 raft_consensus.cc:348] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.437215 32186 raft_consensus.cc:374] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.437245 32186 raft_consensus.cc:729] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efb2818b9cd4c6ea2eeee7d6717b6b1, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.437337 32188 ts_tablet_manager.cc:1428] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:28.437369 32186 consensus_queue.cc:260] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.437628 32186 ts_tablet_manager.cc:1428] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:28.438331 32187 tablet_bootstrap.cc:492] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: No bootstrap required, opened a new log
I20260806 04:29:28.438429 32187 ts_tablet_manager.cc:1397] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:28.438877 32187 raft_consensus.cc:348] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.439018 32187 raft_consensus.cc:374] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:28.439050 32187 raft_consensus.cc:729] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d09cbc82c435f9b3c5ae610613d36, State: Initialized, Role: FOLLOWER
I20260806 04:29:28.439188 32187 consensus_queue.cc:260] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.439451 32187 ts_tablet_manager.cc:1428] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:28.490453 32124 raft_consensus.cc:482] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:28.490634 32124 raft_consensus.cc:504] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.491106 32124 leader_election.cc:288] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:28.491652 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b87e4fed24487afa5cf75b0419f43" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36" is_pre_election: true
I20260806 04:29:28.491675 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b87e4fed24487afa5cf75b0419f43" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" is_pre_election: true
I20260806 04:29:28.491817 31806 raft_consensus.cc:2454] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:28.491835 31983 raft_consensus.cc:2454] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 0.
I20260806 04:29:28.492164 31514 leader_election.cc:302] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94d09cbc82c435f9b3c5ae610613d36, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:28.492367 32124 raft_consensus.cc:2781] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:28.492431 32124 raft_consensus.cc:482] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:28.492457 32124 raft_consensus.cc:3037] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.493594 32124 raft_consensus.cc:504] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.493986 32124 leader_election.cc:288] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Requested vote from peers b94d09cbc82c435f9b3c5ae610613d36 (127.28.94.2:42665), 0efb2818b9cd4c6ea2eeee7d6717b6b1 (127.28.94.3:37863)
I20260806 04:29:28.494429 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b87e4fed24487afa5cf75b0419f43" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94d09cbc82c435f9b3c5ae610613d36"
I20260806 04:29:28.494434 31983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b87e4fed24487afa5cf75b0419f43" candidate_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1"
I20260806 04:29:28.494562 31983 raft_consensus.cc:3037] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.494570 31806 raft_consensus.cc:3037] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:28.495522 31806 raft_consensus.cc:2454] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:28.495532 31983 raft_consensus.cc:2454] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6dbcd51ecb74418a11b58fc93538cdb in term 1.
I20260806 04:29:28.495873 31516 leader_election.cc:302] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efb2818b9cd4c6ea2eeee7d6717b6b1, c6dbcd51ecb74418a11b58fc93538cdb; no voters:
I20260806 04:29:28.496068 32124 raft_consensus.cc:2781] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:28.496171 32124 raft_consensus.cc:686] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Becoming Leader. State: Replica: c6dbcd51ecb74418a11b58fc93538cdb, State: Running, Role: LEADER
I20260806 04:29:28.496327 32124 consensus_queue.cc:237] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } }
I20260806 04:29:28.498169 31393 catalog_manager.cc:5495] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1). New cstate: current_term: 1 leader_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6dbcd51ecb74418a11b58fc93538cdb" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:28.529886 31983 raft_consensus.cc:1261] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:28.529938 31806 raft_consensus.cc:1261] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Refusing update from remote peer c6dbcd51ecb74418a11b58fc93538cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:28.530323 32163 consensus_queue.cc:1035] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efb2818b9cd4c6ea2eeee7d6717b6b1" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:28.530508 32151 consensus_queue.cc:1035] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94d09cbc82c435f9b3c5ae610613d36" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:29:28.543991 32149 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:29:30.104882 31394 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:30.105104 31394 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:30.107673 31394 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a6eb272ebff41fb8b0b8b63202b887d: Sending DeleteTablet for 3 replicas of tablet ba2b87e4fed24487afa5cf75b0419f43
I20260806 04:29:30.108304 31578 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:30 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:42802
I20260806 04:29:30.108305 31777 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:30 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260806 04:29:30.108402 31955 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2b87e4fed24487afa5cf75b0419f43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:30 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:39684
I20260806 04:29:30.108716 32232 tablet_replica.cc:330] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb: stopping tablet replica
I20260806 04:29:30.108796 32235 tablet_replica.cc:330] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: stopping tablet replica
I20260806 04:29:30.108841 32233 tablet_replica.cc:330] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: stopping tablet replica
I20260806 04:29:30.108860 32232 raft_consensus.cc:2229] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:30.108959 32235 raft_consensus.cc:2229] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:30.108982 32233 raft_consensus.cc:2229] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:30.109088 32235 raft_consensus.cc:2258] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:30.109098 32233 raft_consensus.cc:2258] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:30.109091 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 31491
I20260806 04:29:30.109117 32232 raft_consensus.cc:2258] T ba2b87e4fed24487afa5cf75b0419f43 P c6dbcd51ecb74418a11b58fc93538cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:30.116984 32233 ts_tablet_manager.cc:1905] T ba2b87e4fed24487afa5cf75b0419f43 P b94d09cbc82c435f9b3c5ae610613d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:30.119032 32235 ts_tablet_manager.cc:1905] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:29:30.127890 31378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:34465 (error 108)
I20260806 04:29:30.128244 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 31688
I20260806 04:29:30.137099 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:30.132650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.0.0.1:58546)
0806 04:29:30.136853 (+ 4203us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:30.136977 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4095,"thread_start_us":614,"threads_started":1}
W20260806 04:29:30.137253 31378 catalog_manager.cc:4584] TS c6dbcd51ecb74418a11b58fc93538cdb (127.28.94.1:34465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba2b87e4fed24487afa5cf75b0419f43: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34465: connect: Connection refused (error 111)
I20260806 04:29:30.190505 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:30.190128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.0.0.1:58556)
0806 04:29:30.190313 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:30.190387 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:29:30.191226 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:30.191024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34465 (local address 127.0.0.1:58572)
0806 04:29:30.191125 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:30.191173 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:29:30.217815 32235 ts_tablet_manager.cc:1918] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:29:30.217931 32235 log.cc:1196] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/ba2b87e4fed24487afa5cf75b0419f43
I20260806 04:29:30.218194 32235 ts_tablet_manager.cc:1939] T ba2b87e4fed24487afa5cf75b0419f43 P 0efb2818b9cd4c6ea2eeee7d6717b6b1: Deleting consensus metadata
I20260806 04:29:30.224386 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 31875
I20260806 04:29:30.224831 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:30.224505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42665 (local address 127.0.0.1:60966)
0806 04:29:30.224695 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:30.224741 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.2:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260806 04:29:30.242118 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 31350
2026-08-06T04:29:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7274 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:30 dist-test-slave-ncp9 krb5kdc[32240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:30 dist-test-slave-ncp9 krb5kdc[32240](info): set up 2 sockets
Aug 06 04:29:30 dist-test-slave-ncp9 krb5kdc[32240](info): commencing operation
krb5kdc: starting...
W20260806 04:29:31.939446 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.666s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:32.073980 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39767
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:32.198796 32255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:32.199106 32255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:32.199149 32255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:32.203382 32255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:32.203466 32255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:32.203485 32255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:32.203501 32255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:32.203516 32255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:32.209297 32255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:32.210567 32255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:32.211735 32255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:32.221208 32260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.221246 32261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.221400 32255 server_base.cc:1034] running on GCE node
W20260806 04:29:32.221216 32263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.222206 32255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:32.223589 32255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:32.224793 32255 hybrid_clock.cc:648] HybridClock initialized: now 1785990572224771 us; error 37 us; skew 500 ppm
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:32.227824 32255 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:32.228989 32255 webserver.cc:460] Webserver started at http://127.28.94.62:39013/ using document root <none> and password file <none>
I20260806 04:29:32.229612 32255 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:32.229702 32255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:32.229956 32255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:32.232270 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "fcac61c65a2b487f9df9f8a22b836d01"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "5ffa04321386e5a89430dc76289b47aa"
server_key_iv: "0961f277a911bf067bacbc9454b07f5b"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.232929 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "fcac61c65a2b487f9df9f8a22b836d01"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "5ffa04321386e5a89430dc76289b47aa"
server_key_iv: "0961f277a911bf067bacbc9454b07f5b"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.237197 32255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:29:32.239956 32274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.241443 32255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:32.241611 32255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "fcac61c65a2b487f9df9f8a22b836d01"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "5ffa04321386e5a89430dc76289b47aa"
server_key_iv: "0961f277a911bf067bacbc9454b07f5b"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.241750 32255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:32.263854 32255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:32.267005 32255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:32.267218 32255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:32.274976 32255 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39767
I20260806 04:29:32.274987 32351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39767 every 8 connection(s)
I20260806 04:29:32.276064 32255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:32.279024 32352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:32.282476 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 32255
I20260806 04:29:32.282606 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:32.282917 29048 external_mini_cluster.cc:1428] Setting key 75d02e1839accf82be1af65c02b16d80
I20260806 04:29:32.285745 32352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01: Bootstrap starting.
I20260806 04:29:32.288532 32352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:32.289582 32352 log.cc:824] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:32.292193 32352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01: No bootstrap required, opened a new log
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:32.296123 32352 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } }
I20260806 04:29:32.296411 32352 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:32.296481 32352 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcac61c65a2b487f9df9f8a22b836d01, State: Initialized, Role: FOLLOWER
I20260806 04:29:32.297056 32352 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } }
I20260806 04:29:32.297257 32352 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:32.297329 32352 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:32.297427 32352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:32.298317 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:32.284469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.28.94.62:39767)
0806 04:29:32.284993 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:32.285012 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:32.285052 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:29:32.285727 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:32.285745 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:32.286143 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:32.286449 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:32.287392 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.288428 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.289629 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.290258 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.290428 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:32.293227 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:32.293233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:32.293249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:32.293283 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:32.294918 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.295388 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.295394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.295400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.295530 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.295818 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.295821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.295822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.296288 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:32.296497 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:32.296825 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 04:29:32.297197 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":158,"threads_started":1}
I20260806 04:29:32.298830 32352 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } }
I20260806 04:29:32.299201 32352 leader_election.cc:302] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcac61c65a2b487f9df9f8a22b836d01; no voters:
I20260806 04:29:32.299578 32352 leader_election.cc:288] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:32.299687 32358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:32.299943 32358 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [term 1 LEADER]: Becoming Leader. State: Replica: fcac61c65a2b487f9df9f8a22b836d01, State: Running, Role: LEADER
I20260806 04:29:32.300303 32358 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } }
I20260806 04:29:32.301049 32352 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:32.301662 32363 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcac61c65a2b487f9df9f8a22b836d01. Latest consensus state: current_term: 1 leader_uuid: "fcac61c65a2b487f9df9f8a22b836d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } } }
I20260806 04:29:32.301807 32363 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:32.302119 32361 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcac61c65a2b487f9df9f8a22b836d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcac61c65a2b487f9df9f8a22b836d01" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39767 } } }
I20260806 04:29:32.302258 32361 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:32.302371 32369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:32.305861 32369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:32.312412 32369 catalog_manager.cc:1347] Generated new cluster ID: 5d6f2df7eb5e48f1914e4cf282670341
I20260806 04:29:32.312502 32369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:32.321105 32369 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:32.322255 32369 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:32.328504 32369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcac61c65a2b487f9df9f8a22b836d01: Generated new TSK 0
I20260806 04:29:32.329375 32369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:32.362057 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:32.487680 32391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:32.488008 32391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:32.488054 32391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:32.492362 32391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:32.492453 32391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:32.492516 32391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:32.498335 32391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:32.499598 32391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:32.500767 32391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:32.511564 32400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.511570 32402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.511564 32399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.511859 32391 server_base.cc:1034] running on GCE node
I20260806 04:29:32.512420 32391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:32.513192 32391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:32.514367 32391 hybrid_clock.cc:648] HybridClock initialized: now 1785990572514346 us; error 33 us; skew 500 ppm
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:32.517482 32391 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:32.518639 32391 webserver.cc:460] Webserver started at http://127.28.94.1:38211/ using document root <none> and password file <none>
I20260806 04:29:32.519295 32391 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:32.519381 32391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:32.519629 32391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:32.522092 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "e1543f3b02ae439895b1d8cca969ff29"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "505794b608fe6c2007846c4465313643"
server_key_iv: "c01038eb863770d9b429fb4c0b9eebe8"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.522779 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "e1543f3b02ae439895b1d8cca969ff29"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "505794b608fe6c2007846c4465313643"
server_key_iv: "c01038eb863770d9b429fb4c0b9eebe8"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.527380 32391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:32.530439 32413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.531908 32391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:32.532073 32391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "e1543f3b02ae439895b1d8cca969ff29"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "505794b608fe6c2007846c4465313643"
server_key_iv: "c01038eb863770d9b429fb4c0b9eebe8"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.532207 32391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:32.551156 32391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:32.554760 32391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:32.554975 32391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:32.555644 32391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:32.556602 32391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:32.556653 32391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.556713 32391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:32.556740 32391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.565939 32391 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:46515
I20260806 04:29:32.565989 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:46515 every 8 connection(s)
I20260806 04:29:32.566951 32391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:32.570921 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 32391
I20260806 04:29:32.571034 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:32.571285 29048 external_mini_cluster.cc:1428] Setting key 7a7dbe9c22d4460a2dae466e4f1b1c69
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:32.582427 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:32.568965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:48305 (local address 127.28.94.62:39767)
0806 04:29:32.569096 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:32.569100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:32.570063 (+ 963us) server_negotiation.cc:407] Connection header received
0806 04:29:32.571475 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:32.571479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:32.571546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:32.571648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:32.573164 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.573705 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.574975 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.575245 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.575337 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:32.578447 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:32.578453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:32.578456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:32.578493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:32.579897 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.580591 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.580593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.580595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.580671 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.581110 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.581112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.581113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.581310 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:32.581384 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:32.582151 (+ 767us) server_negotiation.cc:299] Negotiation successful
0806 04:29:32.582289 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:32.583576 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:32.569394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39767 (local address 127.28.94.1:48305)
0806 04:29:32.569899 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:32.569947 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:32.571209 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:32.571840 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:32.571848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:32.572351 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:32.573015 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:32.573034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.573838 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.573841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:32.574809 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:32.574823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.575416 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.575420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:32.575648 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:32.576315 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:32.576338 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:32.578209 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:32.580040 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:32.580048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:32.580071 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:32.580478 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:32.580757 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:32.580760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:32.580762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:32.581018 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:32.581417 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:32.581424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:32.581834 (+ 410us) client_negotiation.cc:769] Sending connection context
0806 04:29:32.582087 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 04:29:32.582531 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260806 04:29:32.585078 32561 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:29:32.585394 32561 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:32.586016 32561 heartbeater.cc:511] Master 127.28.94.62:39767 requested a full tablet report, sending...
I20260806 04:29:32.587903 32299 ts_manager.cc:194] Registered new tserver with Master: e1543f3b02ae439895b1d8cca969ff29 (127.28.94.1:46515)
I20260806 04:29:32.589200 32299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:48305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:32.602083 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:32.725718 32566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:32.726043 32566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:32.726109 32566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:32.730405 32566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:32.730492 32566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:32.730587 32566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:32.736481 32566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:32.737764 32566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:32.738893 32566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:32.749403 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.749418 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.749403 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.749670 32566 server_base.cc:1034] running on GCE node
I20260806 04:29:32.750335 32566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:32.751061 32566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:32.752231 32566 hybrid_clock.cc:648] HybridClock initialized: now 1785990572752207 us; error 35 us; skew 500 ppm
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:32.755048 32566 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:32.756067 32566 webserver.cc:460] Webserver started at http://127.28.94.2:36871/ using document root <none> and password file <none>
I20260806 04:29:32.756669 32566 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:32.756742 32566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:32.757022 32566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:32.759282 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "def610fa054e4a8a8238d2d162db43dc"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "d98351966be87157afbc9814da15928b"
server_key_iv: "673d22fccc48de4188573775e5112590"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.759882 32566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "def610fa054e4a8a8238d2d162db43dc"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "d98351966be87157afbc9814da15928b"
server_key_iv: "673d22fccc48de4188573775e5112590"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.764078 32566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:29:32.766884 32588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.768249 32566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:32.768392 32566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "def610fa054e4a8a8238d2d162db43dc"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "d98351966be87157afbc9814da15928b"
server_key_iv: "673d22fccc48de4188573775e5112590"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.768512 32566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:32.789521 32566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:32.792585 32566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:32.792788 32566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:32.793473 32566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:32.794406 32566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:32.794458 32566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.794515 32566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:32.794540 32566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:32.804163 32566 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:40667
I20260806 04:29:32.804189 32736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:40667 every 8 connection(s)
I20260806 04:29:32.805224 32566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:32.810518 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 32566
I20260806 04:29:32.810655 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:32.810961 29048 external_mini_cluster.cc:1428] Setting key f3a97bbc41c25b7d8596b23ef03fb8a1
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:32.819963 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:32.807130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:44651 (local address 127.28.94.62:39767)
0806 04:29:32.807300 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:32.807305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:32.808264 (+ 959us) server_negotiation.cc:407] Connection header received
0806 04:29:32.809585 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:32.809588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:32.809643 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:32.809707 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:32.811206 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.811818 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.813375 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.813600 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.813683 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:32.816314 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:32.816317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:32.816319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:32.816347 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:32.817704 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.818358 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.818363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.818365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.818446 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:32.818794 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:32.818798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:32.818800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:32.818999 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:32.819097 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:32.819654 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 04:29:32.819791 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:29:32.820953 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:32.807591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39767 (local address 127.28.94.2:44651)
0806 04:29:32.808095 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:32.808147 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:32.809359 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:32.809874 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:32.809882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:32.810303 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:32.811048 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:32.811070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.811945 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.811949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:32.813259 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:32.813270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:32.813761 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:32.813765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:32.813965 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:32.814601 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:32.814626 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:32.816127 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:32.817956 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:32.817961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:32.817979 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:32.818256 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:32.818532 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:32.818535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:32.818537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:32.818702 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:32.819103 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:32.819111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:32.819400 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 04:29:32.819600 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 04:29:32.819978 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:29:32.822221 32737 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39767
I20260806 04:29:32.822531 32737 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:32.823105 32737 heartbeater.cc:511] Master 127.28.94.62:39767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:29:32.824466 32299 ts_manager.cc:194] Registered new tserver with Master: def610fa054e4a8a8238d2d162db43dc (127.28.94.2:40667)
I20260806 04:29:32.825135 32299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:44651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:32.840324 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:32.960335 32744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:32.960649 32744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:32.960703 32744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:32.964980 32744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:32.965066 32744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:32.965130 32744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:32.971239 32744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:32.972503 32744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:32.973696 32744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:32.984148 32753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:32.984159 32752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.984364 32744 server_base.cc:1034] running on GCE node
W20260806 04:29:32.984151 32755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:32.984939 32744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:32.985704 32744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:32.986874 32744 hybrid_clock.cc:648] HybridClock initialized: now 1785990572986847 us; error 38 us; skew 500 ppm
Aug 06 04:29:32 dist-test-slave-ncp9 krb5kdc[32240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:32.989899 32744 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:32.990932 32744 webserver.cc:460] Webserver started at http://127.28.94.3:41415/ using document root <none> and password file <none>
I20260806 04:29:32.991545 32744 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:32.991636 32744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:32.991868 32744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:32.994251 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "1164efea06c8428596ed06c59eedbc6a"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "ba42adf029ae9b2a21f6e99a4cb809ff"
server_key_iv: "475e8ed157c88187fee82da1c00a4a38"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.994889 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "1164efea06c8428596ed06c59eedbc6a"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "ba42adf029ae9b2a21f6e99a4cb809ff"
server_key_iv: "475e8ed157c88187fee82da1c00a4a38"
server_key_version: "encryptionkey@0"
I20260806 04:29:32.999347 32744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:33.002337 32762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:33.003772 32744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:33.003914 32744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "1164efea06c8428596ed06c59eedbc6a"
format_stamp: "Formatted at 2026-08-06 04:29:32 on dist-test-slave-ncp9"
server_key: "ba42adf029ae9b2a21f6e99a4cb809ff"
server_key_iv: "475e8ed157c88187fee82da1c00a4a38"
server_key_version: "encryptionkey@0"
I20260806 04:29:33.004037 32744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:33.026924 32744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:33.030462 32744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:33.030674 32744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:33.031339 32744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:33.032469 32744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:33.032528 32744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:33.032591 32744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:33.032621 32744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:33.041975 32744 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:34339
I20260806 04:29:33.041999 439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:34339 every 8 connection(s)
I20260806 04:29:33.043030 32744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:33.048504 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 32744
I20260806 04:29:33.048643 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:33.049016 29048 external_mini_cluster.cc:1428] Setting key 906887da0384b1000bdcc3b0669223d5
Aug 06 04:29:33 dist-test-slave-ncp9 krb5kdc[32240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990572, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:33.057775 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:33.045093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:35903 (local address 127.28.94.62:39767)
0806 04:29:33.045244 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:33.045250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:33.046276 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 04:29:33.047711 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:33.047714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:33.047774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:33.047849 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:33.049547 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:33.050168 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:33.051361 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:33.051585 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:33.051675 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:33.054228 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:33.054231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:33.054233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:33.054263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:33.055506 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:33.056059 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:33.056062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:33.056063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:33.056150 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:33.056509 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:33.056512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:33.056514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:33.056726 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:33.056816 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:33.057446 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 04:29:33.057598 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:33.058686 443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:33.045580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39767 (local address 127.28.94.3:35903)
0806 04:29:33.046117 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:33.046168 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:33.047435 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:33.048041 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:33.048048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:33.048575 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:33.049355 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:33.049380 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:33.050301 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:33.050304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:33.051236 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:33.051247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:33.051739 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:33.051743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:33.051913 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:33.052500 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:33.052518 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:33.054040 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:33.055630 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:33.055637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:33.055658 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:33.055943 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:33.056236 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:33.056239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:33.056241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:33.056402 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:33.056832 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:33.056840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:33.057177 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 04:29:33.057361 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 04:29:33.057776 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":147,"threads_started":1}
I20260806 04:29:33.059984 441 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39767
I20260806 04:29:33.060228 441 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:33.060761 441 heartbeater.cc:511] Master 127.28.94.62:39767 requested a full tablet report, sending...
I20260806 04:29:33.062005 32299 ts_manager.cc:194] Registered new tserver with Master: 1164efea06c8428596ed06c59eedbc6a (127.28.94.3:34339)
I20260806 04:29:33.062651 32299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:35903
I20260806 04:29:33.063158 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:29:33.078944 32356 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:33.077290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.28.94.62:39767)
0806 04:29:33.077440 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:33.077446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:33.077639 (+ 193us) server_negotiation.cc:407] Connection header received
0806 04:29:33.077792 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:33.077795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:33.077846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:33.077915 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:33.078785 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42446: BlockingRecv error: recv got EOF from 127.0.0.1:42446 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:29:33.079965 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 32391
I20260806 04:29:33.098793 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 32566
I20260806 04:29:33.111694 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 32744
I20260806 04:29:33.125790 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 32255
2026-08-06T04:29:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2882 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:33 dist-test-slave-ncp9 krb5kdc[455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:33 dist-test-slave-ncp9 krb5kdc[455](info): set up 2 sockets
Aug 06 04:29:33 dist-test-slave-ncp9 krb5kdc[455](info): commencing operation
krb5kdc: starting...
W20260806 04:29:34.278498 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:34.410292 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43599
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:34.546010 471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:34.546365 471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:34.546420 471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:34.550913 471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:34.551019 471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:34.551043 471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:34.551064 471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:34.551087 471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:34.557149 471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:34.558482 471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:34.559659 471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:34.568852 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:34.568842 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:34.568842 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:34.569288 471 server_base.cc:1034] running on GCE node
I20260806 04:29:34.569926 471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:34.571206 471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:34.572383 471 hybrid_clock.cc:648] HybridClock initialized: now 1785990574572365 us; error 31 us; skew 500 ppm
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990574, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:34.575371 471 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:34.576396 471 webserver.cc:460] Webserver started at http://127.28.94.62:43531/ using document root <none> and password file <none>
I20260806 04:29:34.577044 471 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:34.577121 471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:34.577344 471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:34.579619 471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "6f91617520a84f04be52dd2312ec8da7"
format_stamp: "Formatted at 2026-08-06 04:29:34 on dist-test-slave-ncp9"
server_key: "073de622a35e4ec88380fc4b39945cdd"
server_key_iv: "b7f0fd572217ad55a0aa80615507f320"
server_key_version: "encryptionkey@0"
I20260806 04:29:34.580224 471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "6f91617520a84f04be52dd2312ec8da7"
format_stamp: "Formatted at 2026-08-06 04:29:34 on dist-test-slave-ncp9"
server_key: "073de622a35e4ec88380fc4b39945cdd"
server_key_iv: "b7f0fd572217ad55a0aa80615507f320"
server_key_version: "encryptionkey@0"
I20260806 04:29:34.584430 471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:34.587220 489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:34.588681 471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:34.588825 471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "6f91617520a84f04be52dd2312ec8da7"
format_stamp: "Formatted at 2026-08-06 04:29:34 on dist-test-slave-ncp9"
server_key: "073de622a35e4ec88380fc4b39945cdd"
server_key_iv: "b7f0fd572217ad55a0aa80615507f320"
server_key_version: "encryptionkey@0"
I20260806 04:29:34.588985 471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:34.604241 471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:34.607421 471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:34.607645 471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:34.615557 471 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:43599
I20260806 04:29:34.615816 564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:43599 every 8 connection(s)
I20260806 04:29:34.616674 471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:34.618736 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 471
I20260806 04:29:34.618858 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:34.619124 29048 external_mini_cluster.cc:1428] Setting key 2d17cc08897464e2a9aad66113be76f7
I20260806 04:29:34.619954 565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:34.627996 565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7: Bootstrap starting.
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:34.630983 565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:34.631918 565 log.cc:824] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:34.634574 565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7: No bootstrap required, opened a new log
I20260806 04:29:34.635281 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:34.620506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.28.94.62:43599)
0806 04:29:34.621011 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:34.621032 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:34.621079 (+ 47us) server_negotiation.cc:407] Connection header received
0806 04:29:34.621973 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:34.622001 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:34.622510 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:34.622940 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:34.623730 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:34.625265 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:34.626279 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:34.627053 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:34.627234 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:34.630224 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:34.630230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:34.630246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:34.630285 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:34.632080 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:34.632650 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:34.632658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:34.632666 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:34.632823 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:34.633149 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:34.633152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:34.633153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:34.633597 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:34.633752 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:34.634097 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 04:29:34.634441 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260806 04:29:34.638715 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:29:34.639008 565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } }
I20260806 04:29:34.639283 565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:34.639345 565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91617520a84f04be52dd2312ec8da7, State: Initialized, Role: FOLLOWER
I20260806 04:29:34.639822 565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } }
I20260806 04:29:34.639990 565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:34.640058 565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:34.640169 565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:34.641390 565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } }
I20260806 04:29:34.641845 565 leader_election.cc:302] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f91617520a84f04be52dd2312ec8da7; no voters:
I20260806 04:29:34.642295 565 leader_election.cc:288] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:34.642412 573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:34.642647 573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [term 1 LEADER]: Becoming Leader. State: Replica: 6f91617520a84f04be52dd2312ec8da7, State: Running, Role: LEADER
I20260806 04:29:34.643040 573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } }
I20260806 04:29:34.643857 565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:34.644820 579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f91617520a84f04be52dd2312ec8da7. Latest consensus state: current_term: 1 leader_uuid: "6f91617520a84f04be52dd2312ec8da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } } }
I20260806 04:29:34.644801 577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f91617520a84f04be52dd2312ec8da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f91617520a84f04be52dd2312ec8da7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43599 } } }
I20260806 04:29:34.644991 579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:34.644991 577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:34.645628 587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:34.649363 587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:34.655349 587 catalog_manager.cc:1347] Generated new cluster ID: 7d22778c20f74be69669748f6ce3d268
I20260806 04:29:34.655448 587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:34.675163 587 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:34.676193 587 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:34.679023 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:34.673120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.28.94.62:43599)
0806 04:29:34.673270 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:34.673275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:34.673597 (+ 322us) server_negotiation.cc:407] Connection header received
0806 04:29:34.673737 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:34.673741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:34.673800 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:34.673893 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:34.674626 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:34.675283 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:34.676513 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:34.676834 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:34.676993 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:34.677949 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:34.677953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:34.677956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:34.678532 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:34.678567 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:34.678585 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:34.678681 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:34.678780 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 04:29:34.678867 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:29:34.689812 587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f91617520a84f04be52dd2312ec8da7: Generated new TSK 0
I20260806 04:29:34.690652 587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:29:34.698192 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 471
2026-08-06T04:29:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1573 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[610](info): set up 2 sockets
Aug 06 04:29:34 dist-test-slave-ncp9 krb5kdc[610](info): commencing operation
krb5kdc: starting...
W20260806 04:29:35.814471 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:35 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:35.948176 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45875
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:36.074229 627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:36.074568 627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:36.074617 627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:36.078994 627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:36.079087 627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:36.079105 627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:36.079121 627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:36.079135 627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:36.085086 627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:45875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:36.086315 627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:36.087456 627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:36.096819 634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.096819 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.096832 633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:36.097258 627 server_base.cc:1034] running on GCE node
I20260806 04:29:36.097901 627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:36.099174 627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:36.100353 627 hybrid_clock.cc:648] HybridClock initialized: now 1785990576100330 us; error 39 us; skew 500 ppm
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:36.110714 627 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:36.111874 627 webserver.cc:460] Webserver started at http://127.28.94.62:44289/ using document root <none> and password file <none>
I20260806 04:29:36.112490 627 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:36.112557 627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:36.112766 627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:36.115123 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "e013049cd812467d8dc75275681bc098"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "61bb03e70f9bec7fa1f26c52ad466c33"
server_key_iv: "9961e3ef3f69612f408312c13a26d20d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.115756 627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "e013049cd812467d8dc75275681bc098"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "61bb03e70f9bec7fa1f26c52ad466c33"
server_key_iv: "9961e3ef3f69612f408312c13a26d20d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.120069 627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:36.122936 647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.124369 627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:36.124518 627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "e013049cd812467d8dc75275681bc098"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "61bb03e70f9bec7fa1f26c52ad466c33"
server_key_iv: "9961e3ef3f69612f408312c13a26d20d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.124637 627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:36.147675 627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:36.150854 627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:36.151031 627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:36.158766 627 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:45875
I20260806 04:29:36.159051 716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:45875 every 8 connection(s)
I20260806 04:29:36.159853 627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:36.163015 717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:36.166688 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 627
I20260806 04:29:36.166802 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:36.167047 29048 external_mini_cluster.cc:1428] Setting key 4b9129cd25b1c6558bd84678876c4619
I20260806 04:29:36.169759 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098: Bootstrap starting.
I20260806 04:29:36.172653 717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:36.173754 717 log.cc:824] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:36.176491 717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098: No bootstrap required, opened a new log
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:36.180011 717 raft_consensus.cc:348] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } }
I20260806 04:29:36.180375 717 raft_consensus.cc:374] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:36.180459 717 raft_consensus.cc:729] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e013049cd812467d8dc75275681bc098, State: Initialized, Role: FOLLOWER
I20260806 04:29:36.181042 717 consensus_queue.cc:260] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } }
I20260806 04:29:36.181224 717 raft_consensus.cc:388] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:36.181298 717 raft_consensus.cc:482] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:36.181393 717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:36.182842 717 raft_consensus.cc:504] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } }
I20260806 04:29:36.183243 717 leader_election.cc:302] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e013049cd812467d8dc75275681bc098; no voters:
I20260806 04:29:36.183512 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.168534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.28.94.62:45875)
0806 04:29:36.169021 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:36.169040 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:36.169082 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:29:36.169755 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:36.169774 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:36.170170 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:36.170483 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:36.171462 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.172521 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.173776 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.174379 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.174547 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.177377 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:36.177383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:36.177400 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:36.177434 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:36.179721 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.180324 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.180330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.180335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.180489 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.180817 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.180821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.180823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.181542 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:36.181769 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:36.182037 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 04:29:36.182406 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":127,"threads_started":1}
I20260806 04:29:36.183662 717 leader_election.cc:288] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:36.183888 723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:36.184098 723 raft_consensus.cc:686] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [term 1 LEADER]: Becoming Leader. State: Replica: e013049cd812467d8dc75275681bc098, State: Running, Role: LEADER
I20260806 04:29:36.184455 723 consensus_queue.cc:237] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } }
I20260806 04:29:36.185093 717 sys_catalog.cc:564] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:36.186754 724 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e013049cd812467d8dc75275681bc098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } } }
I20260806 04:29:36.186913 724 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:36.187325 740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:36.187230 726 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e013049cd812467d8dc75275681bc098. Latest consensus state: current_term: 1 leader_uuid: "e013049cd812467d8dc75275681bc098" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013049cd812467d8dc75275681bc098" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45875 } } }
I20260806 04:29:36.187953 726 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:36.191048 740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:36.197621 740 catalog_manager.cc:1347] Generated new cluster ID: e9b981495eb14215a4d893f0f52941f9
I20260806 04:29:36.197708 740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:36.210752 740 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:36.211815 740 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:36.220638 740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e013049cd812467d8dc75275681bc098: Generated new TSK 0
I20260806 04:29:36.221484 740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:36.257431 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:36.408490 750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:36.408803 750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:36.408847 750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:36.413058 750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:36.413146 750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:36.413216 750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:36.418974 750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:36.420229 750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:36.421365 750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:36.432072 760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.432107 757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.432073 755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:36.432437 750 server_base.cc:1034] running on GCE node
I20260806 04:29:36.432989 750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:36.433768 750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:36.434949 750 hybrid_clock.cc:648] HybridClock initialized: now 1785990576434925 us; error 38 us; skew 500 ppm
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:36.438043 750 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:36.439075 750 webserver.cc:460] Webserver started at http://127.28.94.1:41633/ using document root <none> and password file <none>
I20260806 04:29:36.439690 750 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:36.439756 750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:36.439951 750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:36.442272 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "fb6622a7e421458b9ecfc1ae37c6e83d"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "980b5af3b60bbd8c7819dd2cc14cbe7e"
server_key_iv: "4ca1f0694fe283cdd2e1da5daab56b36"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.442930 750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "fb6622a7e421458b9ecfc1ae37c6e83d"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "980b5af3b60bbd8c7819dd2cc14cbe7e"
server_key_iv: "4ca1f0694fe283cdd2e1da5daab56b36"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.447139 750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:36.449858 771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.451237 750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:29:36.451368 750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "fb6622a7e421458b9ecfc1ae37c6e83d"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "980b5af3b60bbd8c7819dd2cc14cbe7e"
server_key_iv: "4ca1f0694fe283cdd2e1da5daab56b36"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.451490 750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:36.473048 750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:36.476429 750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:36.476624 750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:36.477325 750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:36.478220 750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:36.478269 750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.478317 750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:36.478345 750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.487251 750 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:37931
I20260806 04:29:36.487280 923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:37931 every 8 connection(s)
I20260806 04:29:36.488247 750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:36.497242 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 750
I20260806 04:29:36.497380 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:36.497723 29048 external_mini_cluster.cc:1428] Setting key b22170d99c2197a65233f706eb669454
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:36.503237 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.490464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:42277 (local address 127.28.94.62:45875)
0806 04:29:36.490600 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:36.490604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:36.491521 (+ 917us) server_negotiation.cc:407] Connection header received
0806 04:29:36.493011 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:36.493015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:36.493081 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:36.493158 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:36.494452 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.495008 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.496214 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.496448 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.496539 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.499575 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:36.499579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:36.499581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:36.499607 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:36.500867 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.501562 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.501566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.501568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.501672 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.502007 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.502009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.502011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.502283 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:36.502383 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:36.502917 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 04:29:36.503068 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:36.504112 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.490858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45875 (local address 127.28.94.1:42277)
0806 04:29:36.491357 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:36.491405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:36.492770 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:36.493354 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:36.493363 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:36.493761 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:36.494292 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.494307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.495145 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.495148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.496075 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.496086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.496609 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.496613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.496795 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.497519 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:36.497544 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:36.499389 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:36.501012 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.501019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.501039 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.501438 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.501765 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.501767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.501769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.501920 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.502374 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:36.502380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:36.502663 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 04:29:36.502841 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 04:29:36.503140 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260806 04:29:36.505570 925 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45875
I20260806 04:29:36.505889 925 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:36.506623 925 heartbeater.cc:511] Master 127.28.94.62:45875 requested a full tablet report, sending...
I20260806 04:29:36.508459 668 ts_manager.cc:194] Registered new tserver with Master: fb6622a7e421458b9ecfc1ae37c6e83d (127.28.94.1:37931)
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:36.509752 668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:42277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:36.528123 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:36.650007 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:36.650329 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:36.650377 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:36.654580 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:36.654662 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:36.654749 931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:36.660480 931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:36.661815 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:36.662963 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:36.673362 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.673377 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:36.673667 931 server_base.cc:1034] running on GCE node
W20260806 04:29:36.673411 941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:36.674240 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:36.674981 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:36.676164 931 hybrid_clock.cc:648] HybridClock initialized: now 1785990576676142 us; error 42 us; skew 500 ppm
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:36.679349 931 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:36.680472 931 webserver.cc:460] Webserver started at http://127.28.94.2:44605/ using document root <none> and password file <none>
I20260806 04:29:36.681183 931 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:36.681267 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:36.681509 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:36.683876 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "a68294353ab944978a2ebd932fddca35"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "13f2dbf253de4fb6185118a6e3aee127"
server_key_iv: "d0aa8827d7f467081f6428b20db1e36d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.684509 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "a68294353ab944978a2ebd932fddca35"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "13f2dbf253de4fb6185118a6e3aee127"
server_key_iv: "d0aa8827d7f467081f6428b20db1e36d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.689307 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 04:29:36.692072 955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.693571 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:36.693725 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "a68294353ab944978a2ebd932fddca35"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "13f2dbf253de4fb6185118a6e3aee127"
server_key_iv: "d0aa8827d7f467081f6428b20db1e36d"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.693850 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:36.711673 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:36.715462 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:36.715694 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:36.716403 931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:36.717451 931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:36.717521 931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.717585 931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:36.717610 931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.727066 931 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:36933
I20260806 04:29:36.727066 1114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:36933 every 8 connection(s)
I20260806 04:29:36.728212 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:36.736281 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 931
I20260806 04:29:36.736395 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:36.736671 29048 external_mini_cluster.cc:1428] Setting key 39d8f1d879f4659c327b328cc984cb0d
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:36.743629 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.730423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46321 (local address 127.28.94.62:45875)
0806 04:29:36.730563 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:36.730567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:36.731626 (+ 1059us) server_negotiation.cc:407] Connection header received
0806 04:29:36.733264 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:36.733268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:36.733338 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:36.733417 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:36.734800 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.735350 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.736564 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.736822 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.736933 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.739729 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:36.739733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:36.739736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:36.739770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:36.741247 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.741908 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.741910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.741912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.741996 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.742387 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.742389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.742390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.742614 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:36.742694 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:36.743366 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 04:29:36.743496 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:36.745045 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.730880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45875 (local address 127.28.94.2:46321)
0806 04:29:36.731425 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:36.731476 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:36.732982 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:36.733641 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:36.733650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:36.734068 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:36.734625 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.734640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.735482 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.735485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.736434 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.736449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.737036 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.737041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.737269 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.737896 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:36.737916 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:36.739529 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:36.741368 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.741375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.741396 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.741811 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.742077 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.742080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.742082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.742300 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.742716 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:36.742723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:36.743094 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 04:29:36.743338 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 04:29:36.743791 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260806 04:29:36.746389 1115 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45875
I20260806 04:29:36.746711 1115 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:36.747418 1115 heartbeater.cc:511] Master 127.28.94.62:45875 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:29:36.748835 668 ts_manager.cc:194] Registered new tserver with Master: a68294353ab944978a2ebd932fddca35 (127.28.94.2:36933)
I20260806 04:29:36.749547 668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:46321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:36.768368 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:36.891031 1123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:36.891357 1123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:36.891404 1123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:36.895817 1123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:36.895900 1123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:36.895968 1123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:36.901880 1123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:36.903133 1123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:36.904263 1123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:36.915151 1133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.915151 1131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:36.915165 1130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:36.915634 1123 server_base.cc:1034] running on GCE node
I20260806 04:29:36.916128 1123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:36.916885 1123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:36.918087 1123 hybrid_clock.cc:648] HybridClock initialized: now 1785990576918060 us; error 40 us; skew 500 ppm
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:36.921103 1123 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:36.922184 1123 webserver.cc:460] Webserver started at http://127.28.94.3:35845/ using document root <none> and password file <none>
I20260806 04:29:36.922878 1123 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:36.922962 1123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:36.923206 1123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:36.925606 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "8da91060d5c54519bbcf30d794c5f8f1"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "ab2ef224b93919278501113abbf6c36b"
server_key_iv: "e42f3346f70fcda28cad2b2a12a71516"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.926247 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "8da91060d5c54519bbcf30d794c5f8f1"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "ab2ef224b93919278501113abbf6c36b"
server_key_iv: "e42f3346f70fcda28cad2b2a12a71516"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.930717 1123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:36.933596 1143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.935156 1123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:36.935323 1123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "8da91060d5c54519bbcf30d794c5f8f1"
format_stamp: "Formatted at 2026-08-06 04:29:36 on dist-test-slave-ncp9"
server_key: "ab2ef224b93919278501113abbf6c36b"
server_key_iv: "e42f3346f70fcda28cad2b2a12a71516"
server_key_version: "encryptionkey@0"
I20260806 04:29:36.935451 1123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:36.949149 1123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:36.952445 1123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:36.952657 1123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:36.953364 1123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:36.954331 1123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:36.954385 1123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.954443 1123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:36.954473 1123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:36.963792 1123 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:41065
I20260806 04:29:36.963816 1302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:41065 every 8 connection(s)
I20260806 04:29:36.964809 1123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:36.966809 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 1123
I20260806 04:29:36.966951 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:36.967252 29048 external_mini_cluster.cc:1428] Setting key 8104d80e9313330daf2b3b1091dce941
Aug 06 04:29:36 dist-test-slave-ncp9 krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990576, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:36.978536 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.966861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:57249 (local address 127.28.94.62:45875)
0806 04:29:36.967020 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:36.967024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:36.968120 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 04:29:36.969669 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:36.969672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:36.969732 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:36.969803 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:36.971016 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.971555 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.972699 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.972957 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.973060 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.975407 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:36.975411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:36.975414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:36.975441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:36.976526 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.977027 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.977040 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.977042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.977123 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:36.977439 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:36.977443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:36.977445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:36.977637 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:36.977724 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:36.978233 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 04:29:36.978366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:29:36.979431 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:36.967402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45875 (local address 127.28.94.3:57249)
0806 04:29:36.967965 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:36.968021 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:36.969443 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:36.969977 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:36.969985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:36.970358 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:36.970873 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.970886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.971678 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.971681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.972580 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:36.972590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:36.973102 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:36.973107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:36.973274 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:36.973801 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:36.973818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:36.975224 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:36.976632 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.976639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.976655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.976938 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.977205 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:36.977208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:36.977210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:36.977351 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:36.977736 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:36.977744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:36.977997 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 04:29:36.978174 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 04:29:36.978511 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260806 04:29:36.980623 1303 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45875
I20260806 04:29:36.980876 1303 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:36.981467 1303 heartbeater.cc:511] Master 127.28.94.62:45875 requested a full tablet report, sending...
I20260806 04:29:36.982491 668 ts_manager.cc:194] Registered new tserver with Master: 8da91060d5c54519bbcf30d794c5f8f1 (127.28.94.3:41065)
I20260806 04:29:36.983049 668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:57249
I20260806 04:29:36.993114 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:29:37 dist-test-slave-ncp9 krb5kdc[610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 04:29:37 dist-test-slave-ncp9 krb5kdc[610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990577, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:37.021147 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:37.012118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.28.94.62:45875)
0806 04:29:37.012268 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:37.012272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:37.012452 (+ 180us) server_negotiation.cc:407] Connection header received
0806 04:29:37.012623 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:37.012626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:37.012676 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:37.012749 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:37.013549 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:37.014094 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:37.014937 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:37.015158 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:37.015252 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:37.017755 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:37.017759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:37.017761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:37.017796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:37.019339 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:37.019774 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:37.019778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:37.019780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:37.019894 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:37.020168 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:37.020172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:37.020174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:37.020440 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:37.020571 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:37.020820 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:29:37.020977 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 04:29:37.021606 668 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54098
I20260806 04:29:37.022627 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 750
I20260806 04:29:37.040409 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 931
I20260806 04:29:37.053304 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 1123
I20260806 04:29:37.068054 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 627
2026-08-06T04:29:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2369 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:37 dist-test-slave-ncp9 krb5kdc[1318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:37 dist-test-slave-ncp9 krb5kdc[1318](info): set up 2 sockets
Aug 06 04:29:37 dist-test-slave-ncp9 krb5kdc[1318](info): commencing operation
krb5kdc: starting...
W20260806 04:29:38.187282 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:38 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:38.318128 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45029
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:45029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:38.443212 1332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:38.443563 1332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:38.443635 1332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:38.447908 1332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:38.448010 1332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:38.448045 1332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:38.448071 1332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:38.448097 1332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:38.453907 1332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:45029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:38.455160 1332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:38.456292 1332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:38.465379 1338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:38.465381 1339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:38.465736 1332 server_base.cc:1034] running on GCE node
W20260806 04:29:38.465463 1342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:38.466550 1332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:38.467855 1332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:38.469038 1332 hybrid_clock.cc:648] HybridClock initialized: now 1785990578469018 us; error 33 us; skew 500 ppm
Aug 06 04:29:38 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990578, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:38.472085 1332 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:38.473265 1332 webserver.cc:460] Webserver started at http://127.28.94.62:33579/ using document root <none> and password file <none>
I20260806 04:29:38.473894 1332 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:38.473989 1332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:38.474237 1332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:38.476707 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "2b12da6c6ce44ce188d9a39bbcb6286e"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "ddb7495f9bc6e98293f2803612df11a7"
server_key_iv: "32ca03158597d1254bb7275b2b0d2f85"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.477396 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "2b12da6c6ce44ce188d9a39bbcb6286e"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "ddb7495f9bc6e98293f2803612df11a7"
server_key_iv: "32ca03158597d1254bb7275b2b0d2f85"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.481758 1332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:38.484472 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:38.485863 1332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:29:38.486006 1332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "2b12da6c6ce44ce188d9a39bbcb6286e"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "ddb7495f9bc6e98293f2803612df11a7"
server_key_iv: "32ca03158597d1254bb7275b2b0d2f85"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.486135 1332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:38.509159 1332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:38.512550 1332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:38.512759 1332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:38.520427 1332 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:45029
I20260806 04:29:38.520481 1425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:45029 every 8 connection(s)
I20260806 04:29:38.521538 1332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:38.524477 1426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:38.526654 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 1332
I20260806 04:29:38.526765 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:38.527033 29048 external_mini_cluster.cc:1428] Setting key f79d6375b1ecc3a8b9d8aa1c38f53b8d
I20260806 04:29:38.532078 1426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e: Bootstrap starting.
I20260806 04:29:38.534991 1426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:38.535818 1426 log.cc:824] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:29:38 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:38.538350 1426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e: No bootstrap required, opened a new log
I20260806 04:29:38.542852 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:38.528568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.28.94.62:45029)
0806 04:29:38.529058 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:38.529082 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:38.529130 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:29:38.529977 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:38.530005 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:38.530461 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:38.530861 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:38.531510 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.532993 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.533837 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.534557 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.534728 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:38.537313 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:38.537321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:38.537340 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:38.537375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:38.539363 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:38.539812 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:38.539819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:38.539825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:38.539987 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:38.540319 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:38.540323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:38.540325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:38.540960 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:38.541199 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:38.541427 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 04:29:38.541777 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
I20260806 04:29:38.543342 1426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } }
I20260806 04:29:38.543591 1426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:38.543648 1426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b12da6c6ce44ce188d9a39bbcb6286e, State: Initialized, Role: FOLLOWER
I20260806 04:29:38.544132 1426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } }
I20260806 04:29:38.544284 1426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:38.544361 1426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:38.544473 1426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:38.545576 1426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } }
I20260806 04:29:38.545928 1426 leader_election.cc:302] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b12da6c6ce44ce188d9a39bbcb6286e; no voters:
I20260806 04:29:38.546295 1426 leader_election.cc:288] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:38.546383 1433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:38.546617 1433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [term 1 LEADER]: Becoming Leader. State: Replica: 2b12da6c6ce44ce188d9a39bbcb6286e, State: Running, Role: LEADER
I20260806 04:29:38.546976 1433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } }
I20260806 04:29:38.547580 1426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:38.548676 1437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b12da6c6ce44ce188d9a39bbcb6286e. Latest consensus state: current_term: 1 leader_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } } }
I20260806 04:29:38.548830 1437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:38.549270 1444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:38.549252 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12da6c6ce44ce188d9a39bbcb6286e" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45029 } } }
I20260806 04:29:38.549569 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:38.552661 1444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:38.559032 1444 catalog_manager.cc:1347] Generated new cluster ID: cf9c3a1b39ed4279b6d6ae85fa5ec40c
I20260806 04:29:38.559137 1444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:38.568542 1444 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:38.570062 1444 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:38.579113 1444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b12da6c6ce44ce188d9a39bbcb6286e: Generated new TSK 0
I20260806 04:29:38.580138 1444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:38.612103 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:38.740296 1459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:38.740638 1459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:38.740688 1459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:38.745033 1459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:38.745129 1459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:38.745195 1459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:38.751207 1459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:38.752477 1459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:38.753645 1459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:38.766830 1470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:38.766976 1467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:38.766834 1466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:38.767076 1459 server_base.cc:1034] running on GCE node
I20260806 04:29:38.767746 1459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:38.768510 1459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:38.769718 1459 hybrid_clock.cc:648] HybridClock initialized: now 1785990578769672 us; error 68 us; skew 500 ppm
Aug 06 04:29:38 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990578, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:38.772773 1459 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:38.773862 1459 webserver.cc:460] Webserver started at http://127.28.94.1:35163/ using document root <none> and password file <none>
I20260806 04:29:38.774473 1459 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:38.774544 1459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:38.774783 1459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:38.777093 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "5b360d866f194c69a54f3559cbf30998"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "3a50445596f19ff5605c2473f45ea6ab"
server_key_iv: "568768e4c1f80bf8fffb3afb398040aa"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.777716 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "5b360d866f194c69a54f3559cbf30998"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "3a50445596f19ff5605c2473f45ea6ab"
server_key_iv: "568768e4c1f80bf8fffb3afb398040aa"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.782081 1459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:38.784866 1478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:38.786329 1459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:38.786477 1459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "5b360d866f194c69a54f3559cbf30998"
format_stamp: "Formatted at 2026-08-06 04:29:38 on dist-test-slave-ncp9"
server_key: "3a50445596f19ff5605c2473f45ea6ab"
server_key_iv: "568768e4c1f80bf8fffb3afb398040aa"
server_key_version: "encryptionkey@0"
I20260806 04:29:38.786609 1459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:38.797638 1459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:38.800755 1459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:38.800981 1459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:38.801646 1459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:38.802603 1459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:38.802662 1459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:38.802714 1459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:38.802744 1459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:38.811877 1459 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:36377
I20260806 04:29:38.811919 1637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:36377 every 8 connection(s)
I20260806 04:29:38.812885 1459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:38.820705 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 1459
I20260806 04:29:38.820829 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:38.821134 29048 external_mini_cluster.cc:1428] Setting key 107a6e7fbcdbb5df4a760e59de748c81
Aug 06 04:29:38 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990578, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:38.828058 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:38.814966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:49755 (local address 127.28.94.62:45029)
0806 04:29:38.815114 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:38.815118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:38.816077 (+ 959us) server_negotiation.cc:407] Connection header received
0806 04:29:38.817630 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:38.817634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:38.817703 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:38.817779 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:38.819121 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.819676 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.820976 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.821223 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.821324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:38.824178 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:38.824182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:38.824185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:38.824219 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:38.825687 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:38.826362 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:38.826365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:38.826366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:38.826441 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:38.826816 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:38.826819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:38.826820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:38.827028 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:38.827100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:38.827792 (+ 692us) server_negotiation.cc:299] Negotiation successful
0806 04:29:38.827920 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:38.829546 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:38.815381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45029 (local address 127.28.94.1:49755)
0806 04:29:38.815904 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:38.815954 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:38.817371 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:38.817990 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:38.817999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:38.818404 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:38.818940 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:38.818956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.819812 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.819814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:38.820807 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:38.820821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:38.821437 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:38.821442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:38.821654 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:38.822314 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:38.822332 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:38.823969 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:38.825822 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:38.825831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:38.825855 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:38.826266 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:38.826522 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:38.826525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:38.826527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:38.826729 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:38.827128 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:38.827136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:38.827535 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 04:29:38.827768 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:29:38.828210 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260806 04:29:38.831064 1638 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45029
I20260806 04:29:38.831400 1638 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:38.832150 1638 heartbeater.cc:511] Master 127.28.94.62:45029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:38.834198 1377 ts_manager.cc:194] Registered new tserver with Master: 5b360d866f194c69a54f3559cbf30998 (127.28.94.1:36377)
I20260806 04:29:38.835614 1377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:49755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:38.855598 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:38.978322 1643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:38.978634 1643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:38.978680 1643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:38.982932 1643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:38.983014 1643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:38.983078 1643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:38.988729 1643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:38.990000 1643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:38.991185 1643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:39.001775 1652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:39.001775 1649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:39.001778 1648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:39.002012 1643 server_base.cc:1034] running on GCE node
I20260806 04:29:39.002521 1643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:39.003245 1643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:39.004427 1643 hybrid_clock.cc:648] HybridClock initialized: now 1785990579004401 us; error 42 us; skew 500 ppm
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:39.007380 1643 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:39.008396 1643 webserver.cc:460] Webserver started at http://127.28.94.2:33777/ using document root <none> and password file <none>
I20260806 04:29:39.009058 1643 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:39.009136 1643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:39.009361 1643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:39.011608 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "214a8161783e49ce8b604db485f786c9"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "5e9c23ef41dd198a44fefb21f5d56271"
server_key_iv: "1844feaa05c19e5ae7be6723ffba5e9f"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.012233 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "214a8161783e49ce8b604db485f786c9"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "5e9c23ef41dd198a44fefb21f5d56271"
server_key_iv: "1844feaa05c19e5ae7be6723ffba5e9f"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.016418 1643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:39.019212 1664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.020560 1643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:39.020691 1643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "214a8161783e49ce8b604db485f786c9"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "5e9c23ef41dd198a44fefb21f5d56271"
server_key_iv: "1844feaa05c19e5ae7be6723ffba5e9f"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.020804 1643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:39.033537 1643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:39.037134 1643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:39.037333 1643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:39.037987 1643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:39.038949 1643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:39.039000 1643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.039058 1643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:39.039085 1643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.048485 1643 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43513
I20260806 04:29:39.048514 1814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43513 every 8 connection(s)
I20260806 04:29:39.049574 1643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:39.053965 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 1643
I20260806 04:29:39.054096 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:39.054353 29048 external_mini_cluster.cc:1428] Setting key 74b609c56bf733a06ed4d10bdfff485b
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:39.065294 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.051589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:58737 (local address 127.28.94.62:45029)
0806 04:29:39.051730 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:39.051734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:39.052671 (+ 937us) server_negotiation.cc:407] Connection header received
0806 04:29:39.054271 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:39.054275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:39.054341 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:39.054416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:39.055845 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.056569 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.057810 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.058119 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.058205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.061166 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:39.061170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:39.061173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:39.061206 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:39.062871 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.063494 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.063498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.063500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.063609 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.064002 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.064006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.064007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.064230 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:39.064327 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:39.064954 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 04:29:39.065112 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:29:39.066226 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.052011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45029 (local address 127.28.94.2:58737)
0806 04:29:39.052501 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:39.052548 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:39.054024 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:39.054623 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:39.054631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:39.055111 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:39.055679 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:39.055699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.056697 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.056700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:39.057681 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:39.057692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.058288 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.058292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:39.058506 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.059170 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:39.059195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:39.060957 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:39.063031 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:39.063037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:39.063054 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:39.063370 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:39.063708 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:39.063710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:39.063712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:39.063877 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:39.064341 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:39.064350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:39.064648 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 04:29:39.064831 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 04:29:39.065256 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":136,"threads_started":1}
I20260806 04:29:39.067603 1815 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45029
I20260806 04:29:39.067866 1815 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:29:39.068475 1815 heartbeater.cc:511] Master 127.28.94.62:45029 requested a full tablet report, sending...
I20260806 04:29:39.069768 1377 ts_manager.cc:194] Registered new tserver with Master: 214a8161783e49ce8b604db485f786c9 (127.28.94.2:43513)
I20260806 04:29:39.070381 1377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:58737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:39.084354 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:43303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:39.205741 1823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:39.206066 1823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:39.206112 1823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:39.210316 1823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:39.210393 1823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:39.210456 1823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:39.216187 1823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:43303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:39.217458 1823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:39.218539 1823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:39.229246 1829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:39.229249 1832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:39.229249 1828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:39.229331 1823 server_base.cc:1034] running on GCE node
I20260806 04:29:39.230163 1823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:39.230935 1823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:39.232120 1823 hybrid_clock.cc:648] HybridClock initialized: now 1785990579232103 us; error 32 us; skew 500 ppm
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:39.235248 1823 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:39.236323 1823 webserver.cc:460] Webserver started at http://127.28.94.3:38765/ using document root <none> and password file <none>
I20260806 04:29:39.236987 1823 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:39.237066 1823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:39.237325 1823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:39.239653 1823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "3c598f4d2be945808404b90c97ade120"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "2e6d0c442181a1a977a107fe0502ddf4"
server_key_iv: "60fba67bfc41d36e9617ad773fd752d8"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.240293 1823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "3c598f4d2be945808404b90c97ade120"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "2e6d0c442181a1a977a107fe0502ddf4"
server_key_iv: "60fba67bfc41d36e9617ad773fd752d8"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.244704 1823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:29:39.247594 1840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.249020 1823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:39.249162 1823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "3c598f4d2be945808404b90c97ade120"
format_stamp: "Formatted at 2026-08-06 04:29:39 on dist-test-slave-ncp9"
server_key: "2e6d0c442181a1a977a107fe0502ddf4"
server_key_iv: "60fba67bfc41d36e9617ad773fd752d8"
server_key_version: "encryptionkey@0"
I20260806 04:29:39.249277 1823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:39.274015 1823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:39.277179 1823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:39.277385 1823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:39.278059 1823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:39.279012 1823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:39.279066 1823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.279124 1823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:39.279148 1823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:39.288601 1823 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:38019
I20260806 04:29:39.288691 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:38019 every 8 connection(s)
I20260806 04:29:39.289752 1823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:39.292845 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 1823
I20260806 04:29:39.293015 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:39.293365 29048 external_mini_cluster.cc:1428] Setting key 0447266e0bab8b835d8b2dd42f28f7de
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:39.304639 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.291815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:40601 (local address 127.28.94.62:45029)
0806 04:29:39.292008 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:39.292012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:39.292997 (+ 985us) server_negotiation.cc:407] Connection header received
0806 04:29:39.294812 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:39.294816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:39.294882 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:39.294952 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:39.296365 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.296930 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.298142 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.298370 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.298465 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.301325 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:39.301328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:39.301330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:39.301361 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:39.302633 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.303178 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.303180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.303181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.303257 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.303545 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.303548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.303549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.303738 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:39.303814 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:39.304368 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 04:29:39.304498 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 04:29:39.305557 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.292282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45029 (local address 127.28.94.3:40601)
0806 04:29:39.292802 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:39.292851 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:39.294568 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:39.295157 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:39.295166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:39.295650 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:39.296208 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:39.296224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.297058 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.297062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:39.298009 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:39.298020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.298546 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.298551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:39.298748 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.299378 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:39.299397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:39.301087 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:39.302766 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:39.302773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:39.302789 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:39.303090 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:39.303327 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:39.303329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:39.303331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:39.303478 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:39.303813 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:39.303818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:39.304110 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 04:29:39.304283 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:29:39.304584 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
I20260806 04:29:39.307000 1993 heartbeater.cc:347] Connected to a master server at 127.28.94.62:45029
I20260806 04:29:39.307289 1993 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:39.307925 1993 heartbeater.cc:511] Master 127.28.94.62:45029 requested a full tablet report, sending...
I20260806 04:29:39.309088 1376 ts_manager.cc:194] Registered new tserver with Master: 3c598f4d2be945808404b90c97ade120 (127.28.94.3:38019)
I20260806 04:29:39.309707 1376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:40601
I20260806 04:29:39.319728 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:39.348464 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.338913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.28.94.1:36377)
0806 04:29:39.339242 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:39.339249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:39.339277 (+ 28us) server_negotiation.cc:407] Connection header received
0806 04:29:39.339337 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:39.339342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:39.339549 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:39.339684 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:39.340369 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.341205 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.342016 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.342430 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.342516 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.345048 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:39.345064 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:39.345089 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:39.345121 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:39.346634 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.347065 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.347077 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.347079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.347165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.347357 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.347359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.347360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.347693 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:39.347889 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:39.348107 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 04:29:39.348268 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[1318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:39.359869 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:39.351638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.28.94.62:45029)
0806 04:29:39.351795 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:39.351799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:39.351815 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:29:39.351952 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:39.351956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:39.352009 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:39.352085 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:39.352741 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.353407 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.354193 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:39.354471 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:39.354564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:39.356977 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:39.356981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:39.356984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:39.357018 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:39.358191 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.358593 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.358597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.358599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.358687 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:39.358936 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:39.358940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:39.358941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:39.359205 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:39.359337 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:39.359576 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:29:39.359703 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 04:29:39.360333 1376 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51656
I20260806 04:29:39.361013 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 1459
I20260806 04:29:39.372687 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 1643
I20260806 04:29:39.395588 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 1823
I20260806 04:29:39.412751 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 1332
2026-08-06T04:29:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2341 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[2011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[2011](info): set up 2 sockets
Aug 06 04:29:39 dist-test-slave-ncp9 krb5kdc[2011](info): commencing operation
krb5kdc: starting...
W20260806 04:29:40.534068 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:40 dist-test-slave-ncp9 krb5kdc[2011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:40.668646 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37409
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:40.793118 2031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:40.793445 2031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:40.793488 2031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:40.797871 2031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:40.797958 2031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:40.797977 2031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:40.797994 2031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:40.798010 2031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:40.803917 2031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:40.805263 2031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:40.806509 2031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:29:40.815959 2031 server_base.cc:1034] running on GCE node
W20260806 04:29:40.815900 2037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:40.815945 2036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:40.815900 2041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:40.816954 2031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:40.818315 2031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:40.819482 2031 hybrid_clock.cc:648] HybridClock initialized: now 1785990580819461 us; error 49 us; skew 500 ppm
Aug 06 04:29:40 dist-test-slave-ncp9 krb5kdc[2011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:40.822642 2031 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:40.823565 2031 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:40.823648 2031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:40.823879 2031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:40.826295 2031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "88716b6d89254a228dc0de5db4011c9b"
format_stamp: "Formatted at 2026-08-06 04:29:40 on dist-test-slave-ncp9"
server_key: "7a67a90a60ed2a71094103f530176f66"
server_key_iv: "102ead476d7209f60aa297e24b9aaab1"
server_key_version: "encryptionkey@0"
I20260806 04:29:40.826980 2031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "88716b6d89254a228dc0de5db4011c9b"
format_stamp: "Formatted at 2026-08-06 04:29:40 on dist-test-slave-ncp9"
server_key: "7a67a90a60ed2a71094103f530176f66"
server_key_iv: "102ead476d7209f60aa297e24b9aaab1"
server_key_version: "encryptionkey@0"
I20260806 04:29:40.831324 2031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:40.834236 2049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:40.835676 2031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:40.835817 2031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "88716b6d89254a228dc0de5db4011c9b"
format_stamp: "Formatted at 2026-08-06 04:29:40 on dist-test-slave-ncp9"
server_key: "7a67a90a60ed2a71094103f530176f66"
server_key_iv: "102ead476d7209f60aa297e24b9aaab1"
server_key_version: "encryptionkey@0"
I20260806 04:29:40.835948 2031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:40.859264 2031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:40.862636 2031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:40.862836 2031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:40.870639 2031 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:37409
I20260806 04:29:40.870909 2123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:37409 every 8 connection(s)
I20260806 04:29:40.871415 2031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:40.875104 2124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:40.877622 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2031
I20260806 04:29:40.877766 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:40.878063 29048 external_mini_cluster.cc:1428] Setting key 504d83204ac7005b236b29df1a3d454c
I20260806 04:29:40.882882 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: Bootstrap starting.
I20260806 04:29:40.885528 2124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:40.886358 2124 log.cc:824] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:29:40 dist-test-slave-ncp9 krb5kdc[2011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:40.888652 2124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: No bootstrap required, opened a new log
I20260806 04:29:40.892735 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:40.879694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.28.94.62:37409)
0806 04:29:40.880282 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:40.880300 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:40.880345 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:29:40.881108 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:40.881131 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:40.881523 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:40.881870 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:40.882703 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:40.883762 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:40.884663 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:40.885223 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:40.885380 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:40.887909 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:40.887915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:40.887931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:40.887961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:40.889674 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:40.890137 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:40.890143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:40.890148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:40.890275 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:40.890553 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:40.890557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:40.890559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:40.891008 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:40.891239 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:40.891540 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:29:40.891878 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":172,"threads_started":1}
I20260806 04:29:40.892584 2124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } }
I20260806 04:29:40.892849 2124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:40.892925 2124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88716b6d89254a228dc0de5db4011c9b, State: Initialized, Role: FOLLOWER
I20260806 04:29:40.893421 2124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } }
I20260806 04:29:40.893591 2124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:40.893671 2124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:40.893779 2124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:40.895149 2124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } }
I20260806 04:29:40.895560 2124 leader_election.cc:302] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88716b6d89254a228dc0de5db4011c9b; no voters:
I20260806 04:29:40.895936 2124 leader_election.cc:288] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:40.896047 2131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:40.896281 2131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [term 1 LEADER]: Becoming Leader. State: Replica: 88716b6d89254a228dc0de5db4011c9b, State: Running, Role: LEADER
I20260806 04:29:40.896646 2131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } }
I20260806 04:29:40.897439 2124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:40.898718 2133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88716b6d89254a228dc0de5db4011c9b. Latest consensus state: current_term: 1 leader_uuid: "88716b6d89254a228dc0de5db4011c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } } }
I20260806 04:29:40.898900 2133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:40.898718 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88716b6d89254a228dc0de5db4011c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88716b6d89254a228dc0de5db4011c9b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37409 } } }
I20260806 04:29:40.899173 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:40.899461 2142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:40.902748 2142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:40.910149 2142 catalog_manager.cc:1347] Generated new cluster ID: d4f0f1cdfede405fa3058e0f2f5693af
I20260806 04:29:40.910254 2142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:40.926107 2142 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:40.927587 2142 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:40.939954 2142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: Generated new TSK 0
I20260806 04:29:40.941154 2142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:40.972018 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:41.096415 2159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:41.096733 2159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:41.096800 2159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:41.101183 2159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:41.101256 2159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:41.101344 2159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:41.106851 2159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:41.108024 2159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:41.109153 2159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:41.118942 2169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:41.118944 2168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:41.119093 2159 server_base.cc:1034] running on GCE node
W20260806 04:29:41.119243 2172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:41.119673 2159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:41.120361 2159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:41.121528 2159 hybrid_clock.cc:648] HybridClock initialized: now 1785990581121506 us; error 29 us; skew 500 ppm
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:41.127664 2159 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:41.128512 2159 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:41.128592 2159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:41.128813 2159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:41.131032 2159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "8065eebaa24b4c88bc1bb46fb8343d94"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "8713ec132275a27794d9d1856a0f2abc"
server_key_iv: "7e5d15d46c8d44a44f719bd650cf39dd"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.131593 2159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "8065eebaa24b4c88bc1bb46fb8343d94"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "8713ec132275a27794d9d1856a0f2abc"
server_key_iv: "7e5d15d46c8d44a44f719bd650cf39dd"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.135337 2159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:29:41.137809 2184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.139024 2159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:29:41.139139 2159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "8065eebaa24b4c88bc1bb46fb8343d94"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "8713ec132275a27794d9d1856a0f2abc"
server_key_iv: "7e5d15d46c8d44a44f719bd650cf39dd"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.139237 2159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:41.154973 2159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:41.158533 2159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:41.158762 2159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:41.159400 2159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:41.160272 2159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:41.160326 2159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.160368 2159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:41.160388 2159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.168841 2159 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:45335
I20260806 04:29:41.168869 2340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:45335 every 8 connection(s)
I20260806 04:29:41.169486 2159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:41.170532 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2159
I20260806 04:29:41.170648 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:41.170982 29048 external_mini_cluster.cc:1428] Setting key ad39c639085f885dbef3fbaf40250096
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:41.184753 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.171422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:41949 (local address 127.28.94.62:37409)
0806 04:29:41.171597 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.171602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.172606 (+ 1004us) server_negotiation.cc:407] Connection header received
0806 04:29:41.174031 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.174035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.174102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.174177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:41.175830 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.176533 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.177779 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.178150 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.178270 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.181177 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:41.181182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:41.181185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:41.181222 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:41.182767 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.183336 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.183339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.183340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.183417 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.183744 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.183746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.183747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.183929 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:41.183999 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.184482 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.184606 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:29:41.185886 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.171871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37409 (local address 127.28.94.1:41949)
0806 04:29:41.172433 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.172496 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.173812 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.174385 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.174395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.174923 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:41.175648 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.175670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.176674 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.176676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.177640 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.177652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.178345 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.178349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.178527 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.179191 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:41.179208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:41.180977 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:41.182898 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.182904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.182920 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.183239 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.183492 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.183494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.183495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.183671 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.183999 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:41.184005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:41.184277 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.184451 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.184788 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":133,"threads_started":1}
I20260806 04:29:41.187275 2344 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37409
I20260806 04:29:41.187539 2344 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:41.188061 2344 heartbeater.cc:511] Master 127.28.94.62:37409 requested a full tablet report, sending...
I20260806 04:29:41.189898 2073 ts_manager.cc:194] Registered new tserver with Master: 8065eebaa24b4c88bc1bb46fb8343d94 (127.28.94.1:45335)
I20260806 04:29:41.191587 2073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:41949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:41.202160 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:41.320207 2352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:41.320508 2352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:41.320557 2352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:41.324815 2352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:41.324934 2352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:41.325016 2352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:41.331131 2352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:41.332320 2352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:41.333478 2352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:41.343917 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:41.344148 2352 server_base.cc:1034] running on GCE node
W20260806 04:29:41.343917 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:41.343918 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:41.344790 2352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:41.345549 2352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:41.346724 2352 hybrid_clock.cc:648] HybridClock initialized: now 1785990581346694 us; error 40 us; skew 500 ppm
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:41.349717 2352 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:41.350544 2352 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:41.350625 2352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:41.350841 2352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:41.353157 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "a0f2b1717cb8440db08a7d6ab4d32725"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "ddf1221a8a0cefdb9c838007639bed2f"
server_key_iv: "57d93fc2a046bb92660733bb62208aef"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.353760 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "a0f2b1717cb8440db08a7d6ab4d32725"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "ddf1221a8a0cefdb9c838007639bed2f"
server_key_iv: "57d93fc2a046bb92660733bb62208aef"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.358049 2352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:41.360883 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.362402 2352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:41.362578 2352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "a0f2b1717cb8440db08a7d6ab4d32725"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "ddf1221a8a0cefdb9c838007639bed2f"
server_key_iv: "57d93fc2a046bb92660733bb62208aef"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.362709 2352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:41.382997 2352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:41.386574 2352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:41.386795 2352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:41.387485 2352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:41.388458 2352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:41.388509 2352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.388569 2352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:41.388600 2352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.397673 2352 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:34889
I20260806 04:29:41.397699 2542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:34889 every 8 connection(s)
I20260806 04:29:41.398550 2352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:41.400098 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2352
I20260806 04:29:41.400216 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:41.400555 29048 external_mini_cluster.cc:1428] Setting key f7db0830a026c5f1b6a9aa2d49b1c705
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:29:41.413967 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.400597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:56047 (local address 127.28.94.62:37409)
0806 04:29:41.400730 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.400735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.401921 (+ 1186us) server_negotiation.cc:407] Connection header received
0806 04:29:41.403448 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.403452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.403524 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.403611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:41.404988 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.405542 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.406766 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.407005 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.407124 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.409971 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:41.409974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:41.409977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:41.410017 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:41.411607 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.412228 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.412232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.412233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.412318 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.412722 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.412726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.412728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.412997 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:41.413087 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.413683 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.413831 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:41.415205 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.401206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37409 (local address 127.28.94.2:56047)
0806 04:29:41.401766 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.401816 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.403194 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.403825 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.403834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.404249 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:41.404793 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.404808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.405682 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.405685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.406627 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.406638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.407174 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.407179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.407364 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.407972 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:41.407992 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:41.409731 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:41.411768 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.411777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.411797 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.412117 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.412396 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.412398 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.412400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.412572 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.413112 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:41.413118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:41.413409 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.413642 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.414042 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":176,"threads_started":1}
I20260806 04:29:41.416509 2543 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37409
I20260806 04:29:41.416826 2543 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:41.417484 2543 heartbeater.cc:511] Master 127.28.94.62:37409 requested a full tablet report, sending...
I20260806 04:29:41.418658 2073 ts_manager.cc:194] Registered new tserver with Master: a0f2b1717cb8440db08a7d6ab4d32725 (127.28.94.2:34889)
I20260806 04:29:41.419307 2073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:56047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:41.431728 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:41.559072 2551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:41.559397 2551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:41.559446 2551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:41.563678 2551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:41.563768 2551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:41.563834 2551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:41.569751 2551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:41.570982 2551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:41.572057 2551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:41.582530 2559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:41.582536 2564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:41.582537 2562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:41.582830 2551 server_base.cc:1034] running on GCE node
I20260806 04:29:41.583438 2551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:41.584164 2551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:41.585327 2551 hybrid_clock.cc:648] HybridClock initialized: now 1785990581585289 us; error 54 us; skew 500 ppm
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:41.588312 2551 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:41.589195 2551 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:41.589273 2551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:41.589495 2551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:41.591776 2551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "7b8523483d694e75bf61c843027ba179"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "905c7d484ffbc99f733c054ebf4337ec"
server_key_iv: "52848fd847244eb9f9d17baa15e9c216"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.592414 2551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "7b8523483d694e75bf61c843027ba179"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "905c7d484ffbc99f733c054ebf4337ec"
server_key_iv: "52848fd847244eb9f9d17baa15e9c216"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.596596 2551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:29:41.599546 2571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.601050 2551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 04:29:41.601195 2551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "7b8523483d694e75bf61c843027ba179"
format_stamp: "Formatted at 2026-08-06 04:29:41 on dist-test-slave-ncp9"
server_key: "905c7d484ffbc99f733c054ebf4337ec"
server_key_iv: "52848fd847244eb9f9d17baa15e9c216"
server_key_version: "encryptionkey@0"
I20260806 04:29:41.601323 2551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:41.612720 2551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:41.616221 2551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:41.616429 2551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:41.617116 2551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:41.618325 2551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:41.618381 2551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.618427 2551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:41.618445 2551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:41.627614 2551 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:37471
I20260806 04:29:41.627635 2722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:37471 every 8 connection(s)
I20260806 04:29:41.628309 2551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:41.630287 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2551
I20260806 04:29:41.630424 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:41.630731 29048 external_mini_cluster.cc:1428] Setting key ba76576265d1e3b559162f6495691dc6
Aug 06 04:29:41 dist-test-slave-ncp9 krb5kdc[2011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:41.643334 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.630288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:45025 (local address 127.28.94.62:37409)
0806 04:29:41.630433 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.630437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.631596 (+ 1159us) server_negotiation.cc:407] Connection header received
0806 04:29:41.633088 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.633092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.633146 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.633227 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:41.634578 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.635120 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.636341 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.636605 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.636700 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.639467 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:41.639472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:41.639475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:41.639509 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:41.640962 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.641522 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.641527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.641528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.641615 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.641991 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.641995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.641997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.642228 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:41.642321 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.642996 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.643141 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:41.644448 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.630841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37409 (local address 127.28.94.3:45025)
0806 04:29:41.631424 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.631481 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.632823 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.633452 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.633461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.633876 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:41.634415 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.634431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.635256 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.635260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.636204 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.636216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.636784 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.636788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.637022 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.637647 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:41.637675 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:41.639252 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:41.641086 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.641094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.641115 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.641415 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.641711 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:41.641715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:41.641716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:41.641896 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:41.642341 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:41.642349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:41.642663 (+ 314us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.642909 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.643302 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":165,"threads_started":1}
I20260806 04:29:41.645670 2723 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37409
I20260806 04:29:41.645926 2723 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:41.646447 2723 heartbeater.cc:511] Master 127.28.94.62:37409 requested a full tablet report, sending...
I20260806 04:29:41.647567 2072 ts_manager.cc:194] Registered new tserver with Master: 7b8523483d694e75bf61c843027ba179 (127.28.94.3:37471)
I20260806 04:29:41.648175 2072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:45025
I20260806 04:29:41.657699 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:29:41.667554 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.659869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.28.94.62:37409)
0806 04:29:41.660021 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.660026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.660186 (+ 160us) server_negotiation.cc:407] Connection header received
0806 04:29:41.660308 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.660311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.660358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.660428 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:41.661307 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.661845 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.662725 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.662950 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.663046 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.664442 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:41.664446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:41.664448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:41.664478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:41.665749 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.666162 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.666176 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.666177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.666294 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:41.666562 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:41.666566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:41.666567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:41.666829 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:41.666960 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.667251 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.667387 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:29:41.670847 2072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:41.674134 2072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:41.689946 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.684747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.28.94.2:34889)
0806 04:29:41.685269 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.685282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.685306 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:29:41.685382 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.685387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.685564 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.685770 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:41.686108 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687027 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.688385 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.689148 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.689258 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.689432 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.689677 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.689792 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":94,"threads_started":1}
I20260806 04:29:41.689944 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.684236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:34889 (local address 127.0.0.1:38760)
0806 04:29:41.684656 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.684714 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.684862 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.685766 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.685770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.685807 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:41.685988 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.685994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687157 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.687160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.688261 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.688270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.689262 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.689265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.689438 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.689492 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.689610 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.689768 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":57,"spinlock_wait_cycles":2560,"thread_start_us":102,"threads_started":1}
I20260806 04:29:41.690829 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.684679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:45335 (local address 127.0.0.1:45884)
0806 04:29:41.685710 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.685724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.685816 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.686225 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.686228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.686244 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:41.686362 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.686369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687781 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.687784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.689025 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.689037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.690384 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.690388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.690490 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.690517 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.690591 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.690673 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":50,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260806 04:29:41.691267 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.683873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:37471 (local address 127.0.0.1:51354)
0806 04:29:41.684656 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.684714 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.684912 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.685753 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.685760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.685807 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:41.686185 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.686191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687834 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.687837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.689435 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.689444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.690876 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.690879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.690970 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.690993 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.691064 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.691138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":67,"spinlock_wait_cycles":18560,"thread_start_us":92,"threads_started":1}
I20260806 04:29:41.691390 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.684001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.28.94.3:37471)
0806 04:29:41.684481 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.684494 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.684814 (+ 320us) server_negotiation.cc:407] Connection header received
0806 04:29:41.685086 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.685093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.685356 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.685503 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:41.686535 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687701 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.689565 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.690559 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.690653 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.690808 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.691119 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.691232 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":219,"threads_started":1}
I20260806 04:29:41.691924 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.685248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.28.94.1:45335)
0806 04:29:41.685651 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.685659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.685867 (+ 208us) server_negotiation.cc:407] Connection header received
0806 04:29:41.685945 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.685950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.686090 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.686225 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:41.686473 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.687639 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.689188 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.690030 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.690145 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.690312 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.691643 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.691770 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":75,"threads_started":1}
I20260806 04:29:41.694235 2640 tablet_service.cc:1467] Processing CreateTablet for tablet 91f80358fc144586a94663e770da75f6 (DEFAULT_TABLE table=test-table [id=da89ff114b3c461ca78691d08de235dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:41.694482 2443 tablet_service.cc:1467] Processing CreateTablet for tablet 91f80358fc144586a94663e770da75f6 (DEFAULT_TABLE table=test-table [id=da89ff114b3c461ca78691d08de235dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:41.695432 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f80358fc144586a94663e770da75f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:41.695639 2443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f80358fc144586a94663e770da75f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:41.695721 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 91f80358fc144586a94663e770da75f6 (DEFAULT_TABLE table=test-table [id=da89ff114b3c461ca78691d08de235dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:41.696826 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f80358fc144586a94663e770da75f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:41.702881 2749 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Bootstrap starting.
I20260806 04:29:41.703838 2751 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Bootstrap starting.
I20260806 04:29:41.705571 2749 tablet_bootstrap.cc:654] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:41.705823 2753 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: Bootstrap starting.
I20260806 04:29:41.706378 2751 tablet_bootstrap.cc:654] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:41.707760 2751 log.cc:824] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:41.707866 2749 log.cc:824] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:41.708295 2753 tablet_bootstrap.cc:654] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:41.709125 2753 log.cc:824] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:41.709949 2751 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: No bootstrap required, opened a new log
I20260806 04:29:41.710163 2751 ts_tablet_manager.cc:1397] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260806 04:29:41.711054 2753 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: No bootstrap required, opened a new log
I20260806 04:29:41.711258 2753 ts_tablet_manager.cc:1397] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:41.713436 2751 raft_consensus.cc:348] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.713749 2751 raft_consensus.cc:374] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:41.713814 2751 raft_consensus.cc:729] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f2b1717cb8440db08a7d6ab4d32725, State: Initialized, Role: FOLLOWER
I20260806 04:29:41.714370 2751 consensus_queue.cc:260] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.714900 2749 tablet_bootstrap.cc:492] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: No bootstrap required, opened a new log
I20260806 04:29:41.715144 2749 ts_tablet_manager.cc:1397] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260806 04:29:41.715478 2751 ts_tablet_manager.cc:1428] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:29:41.715554 2753 raft_consensus.cc:348] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.715862 2753 raft_consensus.cc:374] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:41.715868 2543 heartbeater.cc:503] Master 127.28.94.62:37409 was elected leader, sending a full tablet report...
I20260806 04:29:41.715919 2753 raft_consensus.cc:729] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8065eebaa24b4c88bc1bb46fb8343d94, State: Initialized, Role: FOLLOWER
I20260806 04:29:41.716483 2753 consensus_queue.cc:260] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.717360 2344 heartbeater.cc:503] Master 127.28.94.62:37409 was elected leader, sending a full tablet report...
I20260806 04:29:41.717690 2753 ts_tablet_manager.cc:1428] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:41.719066 2749 raft_consensus.cc:348] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.719338 2749 raft_consensus.cc:374] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:41.719373 2749 raft_consensus.cc:729] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8523483d694e75bf61c843027ba179, State: Initialized, Role: FOLLOWER
I20260806 04:29:41.719782 2749 consensus_queue.cc:260] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.720865 2723 heartbeater.cc:503] Master 127.28.94.62:37409 was elected leader, sending a full tablet report...
I20260806 04:29:41.721032 2749 ts_tablet_manager.cc:1428] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:29:41.794991 2761 raft_consensus.cc:482] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:41.795238 2761 raft_consensus.cc:504] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.796399 2761 leader_election.cc:288] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b8523483d694e75bf61c843027ba179 (127.28.94.3:37471), a0f2b1717cb8440db08a7d6ab4d32725 (127.28.94.2:34889)
I20260806 04:29:41.800539 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.796821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:46353 (local address 127.28.94.3:37471)
0806 04:29:41.796980 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.796985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.797272 (+ 287us) server_negotiation.cc:407] Connection header received
0806 04:29:41.797413 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.797416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.797466 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.797531 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:41.797800 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.798380 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.799584 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.799990 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.800084 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.800155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.800342 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.800413 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:41.800562 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.796705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:37471 (local address 127.28.94.1:46353)
0806 04:29:41.797171 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.797183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.797307 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.797556 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.797559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.797576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:41.797696 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.797702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.798482 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.798486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.799462 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.799472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.800119 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.800123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.800223 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.800246 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.800311 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.800389 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":1,"thread_start_us":170,"threads_started":1}
I20260806 04:29:41.800719 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.796822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:45197 (local address 127.28.94.2:34889)
0806 04:29:41.796980 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.796985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.797431 (+ 446us) server_negotiation.cc:407] Connection header received
0806 04:29:41.797550 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.797554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.797616 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.797682 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:41.797977 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.798601 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.799778 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.800195 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.800286 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.800349 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.800537 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.800611 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:41.800735 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.796730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:34889 (local address 127.28.94.1:45197)
0806 04:29:41.797330 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.797343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:41.797459 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:41.797708 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:41.797710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:41.797728 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:41.797870 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.797875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.798707 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.798710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.799658 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:41.799668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.800312 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.800316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:41.800424 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.800443 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:29:41.800507 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:29:41.800583 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":19,"thread_start_us":136,"threads_started":1}
I20260806 04:29:41.801258 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f80358fc144586a94663e770da75f6" candidate_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8523483d694e75bf61c843027ba179" is_pre_election: true
I20260806 04:29:41.801456 2470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f80358fc144586a94663e770da75f6" candidate_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" is_pre_election: true
I20260806 04:29:41.801611 2663 raft_consensus.cc:2454] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8065eebaa24b4c88bc1bb46fb8343d94 in term 0.
I20260806 04:29:41.801759 2470 raft_consensus.cc:2454] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8065eebaa24b4c88bc1bb46fb8343d94 in term 0.
I20260806 04:29:41.802148 2188 leader_election.cc:302] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8523483d694e75bf61c843027ba179, 8065eebaa24b4c88bc1bb46fb8343d94; no voters:
I20260806 04:29:41.802461 2761 raft_consensus.cc:2781] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:41.802548 2761 raft_consensus.cc:482] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:41.802583 2761 raft_consensus.cc:3037] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:41.803673 2761 raft_consensus.cc:504] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.804046 2761 leader_election.cc:288] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [CANDIDATE]: Term 1 election: Requested vote from peers 7b8523483d694e75bf61c843027ba179 (127.28.94.3:37471), a0f2b1717cb8440db08a7d6ab4d32725 (127.28.94.2:34889)
I20260806 04:29:41.804453 2663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f80358fc144586a94663e770da75f6" candidate_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b8523483d694e75bf61c843027ba179"
I20260806 04:29:41.804592 2470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f80358fc144586a94663e770da75f6" candidate_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f2b1717cb8440db08a7d6ab4d32725"
I20260806 04:29:41.804584 2663 raft_consensus.cc:3037] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:41.804746 2470 raft_consensus.cc:3037] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:41.805799 2663 raft_consensus.cc:2454] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8065eebaa24b4c88bc1bb46fb8343d94 in term 1.
I20260806 04:29:41.805999 2470 raft_consensus.cc:2454] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8065eebaa24b4c88bc1bb46fb8343d94 in term 1.
I20260806 04:29:41.806171 2188 leader_election.cc:302] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b8523483d694e75bf61c843027ba179, 8065eebaa24b4c88bc1bb46fb8343d94; no voters:
I20260806 04:29:41.806404 2761 raft_consensus.cc:2781] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:41.806787 2761 raft_consensus.cc:686] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [term 1 LEADER]: Becoming Leader. State: Replica: 8065eebaa24b4c88bc1bb46fb8343d94, State: Running, Role: LEADER
I20260806 04:29:41.807214 2761 consensus_queue.cc:237] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } }
I20260806 04:29:41.810801 2073 catalog_manager.cc:5495] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8065eebaa24b4c88bc1bb46fb8343d94 (127.28.94.1). New cstate: current_term: 1 leader_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8065eebaa24b4c88bc1bb46fb8343d94" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:41.845829 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.842036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.28.94.1:45335)
0806 04:29:41.842216 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:41.842220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:41.842238 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:29:41.842374 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:41.842377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:41.842440 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:41.842535 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:41.842797 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.843381 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.844228 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:41.844470 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:41.844559 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:41.844918 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:41.845412 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:41.845528 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:41.845601 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 04:29:41.845675 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:41.852190 2470 raft_consensus.cc:1261] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 1 FOLLOWER]: Refusing update from remote peer 8065eebaa24b4c88bc1bb46fb8343d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:41.852190 2663 raft_consensus.cc:1261] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 1 FOLLOWER]: Refusing update from remote peer 8065eebaa24b4c88bc1bb46fb8343d94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:41.853037 2766 consensus_queue.cc:1035] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b8523483d694e75bf61c843027ba179" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:41.853281 2761 consensus_queue.cc:1035] T 91f80358fc144586a94663e770da75f6 P 8065eebaa24b4c88bc1bb46fb8343d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f2b1717cb8440db08a7d6ab4d32725" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:41.861418 2769 mvcc.cc:205] Tried to move back new op lower bound from 7315417423258705920 to 7315417423085375488. Current Snapshot: MvccSnapshot[applied={T|T < 7315417423258705920}]
I20260806 04:29:41.861661 2771 mvcc.cc:205] Tried to move back new op lower bound from 7315417423258705920 to 7315417423085375488. Current Snapshot: MvccSnapshot[applied={T|T < 7315417423258705920}]
I20260806 04:29:41.873776 2071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:41.873977 2071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:41.876935 2071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88716b6d89254a228dc0de5db4011c9b: Sending DeleteTablet for 3 replicas of tablet 91f80358fc144586a94663e770da75f6
I20260806 04:29:41.878046 2640 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f80358fc144586a94663e770da75f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:41 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:51354
I20260806 04:29:41.878060 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f80358fc144586a94663e770da75f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:41 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45884
I20260806 04:29:41.879118 2443 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f80358fc144586a94663e770da75f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:41 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:38760
I20260806 04:29:41.879482 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2159
I20260806 04:29:41.880177 2783 tablet_replica.cc:330] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: stopping tablet replica
I20260806 04:29:41.882046 2783 raft_consensus.cc:2229] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:41.882519 2783 raft_consensus.cc:2258] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:41.888855 2779 tablet_replica.cc:330] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: stopping tablet replica
I20260806 04:29:41.889297 2779 raft_consensus.cc:2229] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:41.889671 2779 raft_consensus.cc:2258] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:41.891153 2783 ts_tablet_manager.cc:1905] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:41.891152 2779 ts_tablet_manager.cc:1905] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:41.894155 2779 ts_tablet_manager.cc:1918] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:41.894155 2783 ts_tablet_manager.cc:1918] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:41.894282 2783 log.cc:1196] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/91f80358fc144586a94663e770da75f6
I20260806 04:29:41.894591 2783 ts_tablet_manager.cc:1939] T 91f80358fc144586a94663e770da75f6 P a0f2b1717cb8440db08a7d6ab4d32725: Deleting consensus metadata
I20260806 04:29:41.895156 2779 log.cc:1196] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/91f80358fc144586a94663e770da75f6
I20260806 04:29:41.895563 2779 ts_tablet_manager.cc:1939] T 91f80358fc144586a94663e770da75f6 P 7b8523483d694e75bf61c843027ba179: Deleting consensus metadata
I20260806 04:29:41.896255 2056 catalog_manager.cc:4841] TS a0f2b1717cb8440db08a7d6ab4d32725 (127.28.94.2:34889): tablet 91f80358fc144586a94663e770da75f6 (table test-table [id=da89ff114b3c461ca78691d08de235dd]) successfully deleted
I20260806 04:29:41.896919 2052 catalog_manager.cc:4841] TS 7b8523483d694e75bf61c843027ba179 (127.28.94.3:37471): tablet 91f80358fc144586a94663e770da75f6 (table test-table [id=da89ff114b3c461ca78691d08de235dd]) successfully deleted
I20260806 04:29:41.898720 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2352
W20260806 04:29:41.898766 2052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:45335 (error 108)
I20260806 04:29:41.907473 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:41.907062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:45335 (local address 127.0.0.1:45900)
0806 04:29:41.907259 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:41.907356 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:45335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 04:29:41.907698 2052 catalog_manager.cc:4584] TS 8065eebaa24b4c88bc1bb46fb8343d94 (127.28.94.1:45335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91f80358fc144586a94663e770da75f6: Network error: Client connection negotiation failed: client connection to 127.28.94.1:45335: connect: Connection refused (error 111)
I20260806 04:29:41.913129 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2551
I20260806 04:29:41.926265 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2031
2026-08-06T04:29:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2512 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T04:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:41Z Disabled control of system clock
I20260806 04:29:41.963277 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34479
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:29:42.084703 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:42.085068 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:42.085114 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:42.089354 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:42.089426 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:42.089440 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:42.089457 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:42.089471 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:42.094909 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:34479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:42.096107 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:42.097677 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:42.106900 2800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:42.106906 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.107146 2793 server_base.cc:1034] running on GCE node
W20260806 04:29:42.106899 2801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.107985 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:42.109315 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:42.110499 2793 hybrid_clock.cc:648] HybridClock initialized: now 1785990582110481 us; error 37 us; skew 500 ppm
I20260806 04:29:42.112583 2793 webserver.cc:460] Webserver started at http://127.28.94.62:37749/ using document root <none> and password file <none>
I20260806 04:29:42.113286 2793 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:42.113368 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:42.113606 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:42.116097 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "24da59fa51f543ca97538417b402473f"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "b767f81eb3cbef8bfda477955bdc5348"
server_key_iv: "4ceb022e1a5ccbd9fbd12ef45c67ed4d"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.116696 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "24da59fa51f543ca97538417b402473f"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "b767f81eb3cbef8bfda477955bdc5348"
server_key_iv: "4ceb022e1a5ccbd9fbd12ef45c67ed4d"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.120771 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:29:42.123641 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.125077 2793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:42.125252 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "24da59fa51f543ca97538417b402473f"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "b767f81eb3cbef8bfda477955bdc5348"
server_key_iv: "4ceb022e1a5ccbd9fbd12ef45c67ed4d"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.125382 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:42.130414 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:42.131140 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:42.131331 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:42.138665 2793 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:34479
I20260806 04:29:42.138674 2893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:34479 every 8 connection(s)
I20260806 04:29:42.139750 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:42.140116 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2793
I20260806 04:29:42.140224 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:42.140558 29048 external_mini_cluster.cc:1428] Setting key 9d4dd23499e1c5a1d78e5dbf71f67962
I20260806 04:29:42.143334 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:42.147128 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.142113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.28.94.62:34479)
0806 04:29:42.143274 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.143290 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.143330 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:29:42.144087 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.144108 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.144502 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.144814 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.145757 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.145762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.145775 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.145786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.145844 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.145860 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.145982 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.146094 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.146355 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":589,"threads_started":1}
I20260806 04:29:42.150056 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: Bootstrap starting.
I20260806 04:29:42.152638 2894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:42.153503 2894 log.cc:824] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:42.155793 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: No bootstrap required, opened a new log
I20260806 04:29:42.159873 2894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } }
I20260806 04:29:42.160180 2894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:42.160264 2894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24da59fa51f543ca97538417b402473f, State: Initialized, Role: FOLLOWER
I20260806 04:29:42.160822 2894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } }
I20260806 04:29:42.161029 2894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:42.161109 2894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:42.161226 2894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:42.162652 2894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } }
I20260806 04:29:42.163038 2894 leader_election.cc:302] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24da59fa51f543ca97538417b402473f; no voters:
I20260806 04:29:42.163399 2894 leader_election.cc:288] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:42.163508 2902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:42.163749 2902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [term 1 LEADER]: Becoming Leader. State: Replica: 24da59fa51f543ca97538417b402473f, State: Running, Role: LEADER
I20260806 04:29:42.164104 2902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } }
I20260806 04:29:42.164783 2894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:42.165904 2905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24da59fa51f543ca97538417b402473f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } } }
I20260806 04:29:42.165905 2906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24da59fa51f543ca97538417b402473f. Latest consensus state: current_term: 1 leader_uuid: "24da59fa51f543ca97538417b402473f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24da59fa51f543ca97538417b402473f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34479 } } }
I20260806 04:29:42.166066 2906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:42.166066 2905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:42.166477 2914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:42.169806 2914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:42.176429 2914 catalog_manager.cc:1347] Generated new cluster ID: c4d7273f38e747dfab09bfe73f5be807
I20260806 04:29:42.176537 2914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:42.199230 2914 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:42.199621 2914 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:42.205511 2914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: Generated new TSK 0
I20260806 04:29:42.206319 2914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:29:42.209931 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--builtin_ntp_servers=127.28.94.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:29:42.334481 2926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:42.334817 2926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:42.334859 2926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:42.339075 2926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:42.339157 2926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:42.339218 2926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:42.344403 2926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:42.345698 2926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:42.347193 2926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:42.357617 2936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:42.357617 2934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.357722 2926 server_base.cc:1034] running on GCE node
W20260806 04:29:42.357928 2932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.358404 2926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:42.359110 2926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:42.360296 2926 hybrid_clock.cc:648] HybridClock initialized: now 1785990582360271 us; error 35 us; skew 500 ppm
I20260806 04:29:42.362392 2926 webserver.cc:460] Webserver started at http://127.28.94.1:38381/ using document root <none> and password file <none>
I20260806 04:29:42.363029 2926 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:42.363117 2926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:42.363346 2926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:42.365748 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "6eac6bff44d542bd9ea631f877c97e54"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "7572ec508b15b1f05b1c5cda8aa59eab"
server_key_iv: "421f0af4a9a02684133b354e48932a7f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.366400 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "6eac6bff44d542bd9ea631f877c97e54"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "7572ec508b15b1f05b1c5cda8aa59eab"
server_key_iv: "421f0af4a9a02684133b354e48932a7f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.370903 2926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 04:29:42.373832 2943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.375208 2926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:29:42.375342 2926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "6eac6bff44d542bd9ea631f877c97e54"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "7572ec508b15b1f05b1c5cda8aa59eab"
server_key_iv: "421f0af4a9a02684133b354e48932a7f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.375470 2926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:42.380524 2926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:42.381300 2926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:42.381479 2926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:42.382083 2926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:42.383147 2926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:42.383196 2926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.383251 2926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:42.383278 2926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.392083 2926 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44865
I20260806 04:29:42.392127 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44865 every 8 connection(s)
I20260806 04:29:42.393114 2926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:42.398422 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 2926
I20260806 04:29:42.398561 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:42.398840 29048 external_mini_cluster.cc:1428] Setting key 5f58c67aa13f9bda713676f0a08fb481
I20260806 04:29:42.400008 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.395184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:52157 (local address 127.28.94.62:34479)
0806 04:29:42.395328 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.395332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.396563 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 04:29:42.398129 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.398133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.398191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.398274 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.399503 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.399507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.399509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.399524 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.399547 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.399556 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.399666 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.399821 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.399865 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:29:42.401069 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--builtin_ntp_servers=127.28.94.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:29:42.401230 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.395634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:34479 (local address 127.28.94.1:52157)
0806 04:29:42.396384 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.396432 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.397865 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.398444 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.398452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.398553 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.399199 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.399200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.399249 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.399331 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.399680 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.399689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.399695 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.399851 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.400109 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
I20260806 04:29:42.402688 3101 heartbeater.cc:347] Connected to a master server at 127.28.94.62:34479
I20260806 04:29:42.402987 3101 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:42.403541 3101 heartbeater.cc:511] Master 127.28.94.62:34479 requested a full tablet report, sending...
I20260806 04:29:42.405339 2839 ts_manager.cc:194] Registered new tserver with Master: 6eac6bff44d542bd9ea631f877c97e54 (127.28.94.1:44865)
W20260806 04:29:42.524425 3105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:42.524772 3105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:42.524832 3105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:42.529116 3105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:42.529191 3105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:42.529274 3105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:42.534427 3105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:42.535712 3105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:42.537293 3105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:42.547726 3110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:42.547729 3111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.548112 3105 server_base.cc:1034] running on GCE node
W20260806 04:29:42.548192 3113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.548657 3105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:42.549409 3105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:42.550588 3105 hybrid_clock.cc:648] HybridClock initialized: now 1785990582550562 us; error 35 us; skew 500 ppm
I20260806 04:29:42.552603 3105 webserver.cc:460] Webserver started at http://127.28.94.2:38107/ using document root <none> and password file <none>
I20260806 04:29:42.553292 3105 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:42.553370 3105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:42.553614 3105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:42.555874 3105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "2555ee4da3ce4c41a96db6087a378b28"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "8701ef295f3db19150d7988057ba5517"
server_key_iv: "3b1efbb2dda1bd8ead89ab7c98e049f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.556564 3105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "2555ee4da3ce4c41a96db6087a378b28"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "8701ef295f3db19150d7988057ba5517"
server_key_iv: "3b1efbb2dda1bd8ead89ab7c98e049f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.560773 3105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:29:42.563694 3124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.565187 3105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:42.565323 3105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "2555ee4da3ce4c41a96db6087a378b28"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "8701ef295f3db19150d7988057ba5517"
server_key_iv: "3b1efbb2dda1bd8ead89ab7c98e049f3"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.565466 3105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:42.570567 3105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:42.571297 3105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:42.571480 3105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:42.572105 3105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:42.573177 3105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:42.573231 3105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.573288 3105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:42.573314 3105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.582700 3105 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:41559
I20260806 04:29:42.582773 3276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:41559 every 8 connection(s)
I20260806 04:29:42.583750 3105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:42.589396 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3105
I20260806 04:29:42.589545 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:42.589859 29048 external_mini_cluster.cc:1428] Setting key ad2bc50375179bbb7afdb2aa7d907f3d
I20260806 04:29:42.590556 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.585879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:55067 (local address 127.28.94.62:34479)
0806 04:29:42.586040 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.586044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.587224 (+ 1180us) server_negotiation.cc:407] Connection header received
0806 04:29:42.588553 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.588556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.588601 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.588680 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.590067 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.590071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.590072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.590088 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.590113 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.590122 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.590210 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.590358 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.590411 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:29:42.591712 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.586302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:34479 (local address 127.28.94.2:55067)
0806 04:29:42.587045 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.587092 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.588348 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.588859 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.588867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.589013 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.589725 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.589727 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.589774 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.589854 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.590203 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.590214 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.590222 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.590404 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.590741 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260806 04:29:42.592047 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--builtin_ntp_servers=127.28.94.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:29:42.593091 3278 heartbeater.cc:347] Connected to a master server at 127.28.94.62:34479
I20260806 04:29:42.593356 3278 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:42.593827 3278 heartbeater.cc:511] Master 127.28.94.62:34479 requested a full tablet report, sending...
I20260806 04:29:42.594993 2839 ts_manager.cc:194] Registered new tserver with Master: 2555ee4da3ce4c41a96db6087a378b28 (127.28.94.2:41559)
W20260806 04:29:42.716580 3281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:42.716913 3281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:42.716970 3281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:42.721282 3281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:42.721357 3281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:42.721428 3281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:42.726634 3281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:34479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:42.727896 3281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:42.729532 3281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:42.740057 3293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:42.740057 3294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.740302 3281 server_base.cc:1034] running on GCE node
W20260806 04:29:42.740085 3296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:42.740854 3281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:42.741621 3281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:42.742812 3281 hybrid_clock.cc:648] HybridClock initialized: now 1785990582742790 us; error 37 us; skew 500 ppm
I20260806 04:29:42.745031 3281 webserver.cc:460] Webserver started at http://127.28.94.3:46339/ using document root <none> and password file <none>
I20260806 04:29:42.745714 3281 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:42.745791 3281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:42.746016 3281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:42.748389 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "8e2ebe41f19d468d847ce357aec49ad8"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "0de56b00e34497eb29583857c7b5aa1b"
server_key_iv: "9dfa44c05271c668b8cca2c5a716452f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.749044 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "8e2ebe41f19d468d847ce357aec49ad8"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "0de56b00e34497eb29583857c7b5aa1b"
server_key_iv: "9dfa44c05271c668b8cca2c5a716452f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.753273 3281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:29:42.756265 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.757764 3281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:42.757913 3281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "8e2ebe41f19d468d847ce357aec49ad8"
format_stamp: "Formatted at 2026-08-06 04:29:42 on dist-test-slave-ncp9"
server_key: "0de56b00e34497eb29583857c7b5aa1b"
server_key_iv: "9dfa44c05271c668b8cca2c5a716452f"
server_key_version: "encryptionkey@0"
I20260806 04:29:42.758057 3281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:42.763043 3281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:42.763736 3281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:42.763895 3281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:42.764485 3281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:42.765724 3281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:42.765781 3281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.765825 3281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:42.765838 3281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:42.775007 3281 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:38747
I20260806 04:29:42.775027 3453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:38747 every 8 connection(s)
I20260806 04:29:42.776026 3281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:42.780539 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3281
I20260806 04:29:42.780687 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:42.781055 29048 external_mini_cluster.cc:1428] Setting key 27cf412ac96ebdc10372127ded9f8031
I20260806 04:29:42.782958 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.778051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:39639 (local address 127.28.94.62:34479)
0806 04:29:42.778199 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.778203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.779445 (+ 1242us) server_negotiation.cc:407] Connection header received
0806 04:29:42.781057 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.781060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.781108 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.781204 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.782511 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.782515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.782517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.782530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.782554 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.782562 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.782642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.782780 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.782825 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:42.784042 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.778483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:34479 (local address 127.28.94.3:39639)
0806 04:29:42.779278 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.779330 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.780751 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.781406 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.781416 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.781521 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.782206 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.782209 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.782269 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.782370 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.782640 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.782649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.782656 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.782811 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.783039 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":184,"threads_started":1}
I20260806 04:29:42.785207 3454 heartbeater.cc:347] Connected to a master server at 127.28.94.62:34479
I20260806 04:29:42.785451 3454 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:42.785915 3454 heartbeater.cc:511] Master 127.28.94.62:34479 requested a full tablet report, sending...
I20260806 04:29:42.787004 2839 ts_manager.cc:194] Registered new tserver with Master: 8e2ebe41f19d468d847ce357aec49ad8 (127.28.94.3:38747)
I20260806 04:29:42.795718 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:29:42.799775 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.798014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.28.94.62:34479)
0806 04:29:42.798153 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.798159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.798371 (+ 212us) server_negotiation.cc:407] Connection header received
0806 04:29:42.798507 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.798510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.798552 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.798625 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.799359 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.799364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.799366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.799379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.799400 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.799408 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.799483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.799587 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.799634 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 04:29:42.802737 2839 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:42.805932 2839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:42.818689 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.816579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.28.94.1:44865)
0806 04:29:42.817062 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.817073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.817095 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:29:42.817145 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.817149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.817321 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.817458 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.818145 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.818152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.818164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.818172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.818206 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.818224 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.818319 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.818463 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.818541 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":113,"threads_started":1}
I20260806 04:29:42.818673 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.816095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44865 (local address 127.0.0.1:42280)
0806 04:29:42.816525 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.816538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.816625 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.817438 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.817441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.817455 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.818002 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.818003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.818012 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.818032 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.818308 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.818318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.818324 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.818432 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.818508 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":103,"threads_started":1}
I20260806 04:29:42.818861 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.815314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38747 (local address 127.0.0.1:43612)
0806 04:29:42.815697 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.815748 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.815857 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.817022 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.817028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.817115 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.817734 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.817736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.817759 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.817838 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.818552 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.818555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.818559 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.818624 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.818678 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
I20260806 04:29:42.819268 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.815532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:41559 (local address 127.0.0.1:36478)
0806 04:29:42.816160 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:42.816175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:42.816261 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:42.817299 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:42.817302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:42.817319 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:42.817868 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:42.817870 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:42.817881 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:42.817913 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:42.818552 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:42.818555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:42.818559 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:29:42.818628 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 04:29:42.818692 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":21,"thread_start_us":132,"threads_started":1}
I20260806 04:29:42.819295 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.815490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.28.94.3:38747)
0806 04:29:42.816512 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.816525 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.816554 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:29:42.816612 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.816618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.816842 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.817015 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.818353 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.818358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.818370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.818380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.818421 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.818442 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.819014 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.819066 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.819155 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":174,"threads_started":1}
I20260806 04:29:42.820205 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:42.816093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.28.94.2:41559)
0806 04:29:42.816859 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:42.816870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:42.816919 (+ 49us) server_negotiation.cc:407] Connection header received
0806 04:29:42.816975 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:42.816981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:42.817147 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:42.817543 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:42.818353 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:42.818358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:42.818370 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:42.818380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:42.818418 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:42.818442 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:42.819928 (+ 1486us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:42.819980 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 04:29:42.820055 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":134,"threads_started":1}
I20260806 04:29:42.822314 3015 tablet_service.cc:1467] Processing CreateTablet for tablet f1cea9ce596b479fa4a996c2358b3545 (DEFAULT_TABLE table=test-table [id=0e18035224bd43909712a427f6b7f178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:42.822314 3368 tablet_service.cc:1467] Processing CreateTablet for tablet f1cea9ce596b479fa4a996c2358b3545 (DEFAULT_TABLE table=test-table [id=0e18035224bd43909712a427f6b7f178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:42.822599 3185 tablet_service.cc:1467] Processing CreateTablet for tablet f1cea9ce596b479fa4a996c2358b3545 (DEFAULT_TABLE table=test-table [id=0e18035224bd43909712a427f6b7f178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:42.823608 3015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cea9ce596b479fa4a996c2358b3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:42.823608 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cea9ce596b479fa4a996c2358b3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:42.823704 3185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cea9ce596b479fa4a996c2358b3545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:42.830847 3482 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Bootstrap starting.
I20260806 04:29:42.831902 3487 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Bootstrap starting.
I20260806 04:29:42.833266 3488 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Bootstrap starting.
I20260806 04:29:42.834035 3482 tablet_bootstrap.cc:654] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:42.834188 3487 tablet_bootstrap.cc:654] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:42.834892 3482 log.cc:824] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:42.835081 3487 log.cc:824] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:42.835580 3488 tablet_bootstrap.cc:654] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:42.836382 3488 log.cc:824] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:42.837270 3482 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: No bootstrap required, opened a new log
I20260806 04:29:42.837291 3487 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: No bootstrap required, opened a new log
I20260806 04:29:42.837488 3487 ts_tablet_manager.cc:1397] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:29:42.837493 3482 ts_tablet_manager.cc:1397] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:29:42.838585 3488 tablet_bootstrap.cc:492] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: No bootstrap required, opened a new log
I20260806 04:29:42.838780 3488 ts_tablet_manager.cc:1397] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:42.841559 3487 raft_consensus.cc:348] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.841825 3487 raft_consensus.cc:374] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:42.841893 3487 raft_consensus.cc:729] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eac6bff44d542bd9ea631f877c97e54, State: Initialized, Role: FOLLOWER
I20260806 04:29:42.842386 3487 consensus_queue.cc:260] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.842254 3482 raft_consensus.cc:348] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.842525 3482 raft_consensus.cc:374] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:42.842592 3482 raft_consensus.cc:729] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555ee4da3ce4c41a96db6087a378b28, State: Initialized, Role: FOLLOWER
I20260806 04:29:42.843035 3482 consensus_queue.cc:260] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.843149 3101 heartbeater.cc:503] Master 127.28.94.62:34479 was elected leader, sending a full tablet report...
I20260806 04:29:42.843318 3488 raft_consensus.cc:348] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.843559 3487 ts_tablet_manager.cc:1428] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:29:42.843581 3488 raft_consensus.cc:374] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:42.843641 3488 raft_consensus.cc:729] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2ebe41f19d468d847ce357aec49ad8, State: Initialized, Role: FOLLOWER
I20260806 04:29:42.844072 3482 ts_tablet_manager.cc:1428] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:29:42.844102 3488 consensus_queue.cc:260] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:42.844588 3278 heartbeater.cc:503] Master 127.28.94.62:34479 was elected leader, sending a full tablet report...
I20260806 04:29:42.844736 3454 heartbeater.cc:503] Master 127.28.94.62:34479 was elected leader, sending a full tablet report...
I20260806 04:29:42.846732 3488 ts_tablet_manager.cc:1428] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260806 04:29:42.894479 3102 tablet.cc:2369] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:43.027127 3457 tablet.cc:2369] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:43.031973 3495 raft_consensus.cc:482] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:43.032204 3495 raft_consensus.cc:504] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:43.033507 3495 leader_election.cc:288] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eac6bff44d542bd9ea631f877c97e54 (127.28.94.1:44865), 2555ee4da3ce4c41a96db6087a378b28 (127.28.94.2:41559)
I20260806 04:29:43.035494 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.033917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:50535 (local address 127.28.94.1:44865)
0806 04:29:43.034049 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.034054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.034375 (+ 321us) server_negotiation.cc:407] Connection header received
0806 04:29:43.034432 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.034435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.034476 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.034538 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:43.035134 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:43.035137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:43.035140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:43.035152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:43.035174 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:43.035184 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:43.035264 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:43.035323 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 04:29:43.035361 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:29:43.035494 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.033993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:44963 (local address 127.28.94.2:41559)
0806 04:29:43.034139 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.034143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.034163 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:29:43.034270 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.034272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.034310 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.034398 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:43.035071 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:43.035076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:43.035079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:43.035102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:43.035125 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:43.035135 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:43.035226 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:43.035310 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 04:29:43.035353 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:29:43.035494 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.033833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:41559 (local address 127.28.94.3:44963)
0806 04:29:43.034047 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:43.034073 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:43.034166 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:43.034434 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:43.034437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:43.034451 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:43.034894 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:43.034895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:43.034908 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:43.034940 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:43.035222 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:43.035225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:43.035228 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:29:43.035304 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:29:43.035344 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260806 04:29:43.035509 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.033838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44865 (local address 127.28.94.3:50535)
0806 04:29:43.034275 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:43.034286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:43.034356 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:43.034584 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:43.034586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:43.034600 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:43.034981 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:29:43.034983 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:43.034992 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:29:43.035014 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:43.035253 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:43.035256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:43.035259 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:29:43.035321 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 04:29:43.035344 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":60,"thread_start_us":147,"threads_started":1}
I20260806 04:29:43.036255 3041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cea9ce596b479fa4a996c2358b3545" candidate_uuid: "8e2ebe41f19d468d847ce357aec49ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eac6bff44d542bd9ea631f877c97e54" is_pre_election: true
I20260806 04:29:43.036270 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cea9ce596b479fa4a996c2358b3545" candidate_uuid: "8e2ebe41f19d468d847ce357aec49ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555ee4da3ce4c41a96db6087a378b28" is_pre_election: true
I20260806 04:29:43.036582 3209 raft_consensus.cc:2454] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2ebe41f19d468d847ce357aec49ad8 in term 0.
I20260806 04:29:43.036583 3041 raft_consensus.cc:2454] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2ebe41f19d468d847ce357aec49ad8 in term 0.
I20260806 04:29:43.037112 3309 leader_election.cc:302] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2555ee4da3ce4c41a96db6087a378b28, 8e2ebe41f19d468d847ce357aec49ad8; no voters:
I20260806 04:29:43.037412 3495 raft_consensus.cc:2781] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:43.037499 3495 raft_consensus.cc:482] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:43.037531 3495 raft_consensus.cc:3037] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:43.038681 3495 raft_consensus.cc:504] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:43.039073 3495 leader_election.cc:288] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 6eac6bff44d542bd9ea631f877c97e54 (127.28.94.1:44865), 2555ee4da3ce4c41a96db6087a378b28 (127.28.94.2:41559)
I20260806 04:29:43.039496 3041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cea9ce596b479fa4a996c2358b3545" candidate_uuid: "8e2ebe41f19d468d847ce357aec49ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eac6bff44d542bd9ea631f877c97e54"
I20260806 04:29:43.039633 3041 raft_consensus.cc:3037] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:43.040741 3041 raft_consensus.cc:2454] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e2ebe41f19d468d847ce357aec49ad8 in term 1.
I20260806 04:29:43.041054 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cea9ce596b479fa4a996c2358b3545" candidate_uuid: "8e2ebe41f19d468d847ce357aec49ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2555ee4da3ce4c41a96db6087a378b28"
I20260806 04:29:43.041105 3306 leader_election.cc:302] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eac6bff44d542bd9ea631f877c97e54, 8e2ebe41f19d468d847ce357aec49ad8; no voters:
I20260806 04:29:43.041198 3209 raft_consensus.cc:3037] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:43.041301 3495 raft_consensus.cc:2781] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:43.041563 3495 raft_consensus.cc:686] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2ebe41f19d468d847ce357aec49ad8, State: Running, Role: LEADER
I20260806 04:29:43.041981 3495 consensus_queue.cc:237] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } }
I20260806 04:29:43.042585 3209 raft_consensus.cc:2454] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e2ebe41f19d468d847ce357aec49ad8 in term 1.
I20260806 04:29:43.045759 2839 catalog_manager.cc:5495] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e2ebe41f19d468d847ce357aec49ad8 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "8e2ebe41f19d468d847ce357aec49ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e2ebe41f19d468d847ce357aec49ad8" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:43.057822 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.056459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.28.94.3:38747)
0806 04:29:43.056583 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.056587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.056611 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:29:43.056743 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.056745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.056809 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.056877 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:43.057485 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:43.057488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:43.057490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:43.057501 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:43.057521 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:43.057530 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:43.057609 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:43.057642 (+ 33us) server_negotiation.cc:299] Negotiation successful
0806 04:29:43.057679 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:29:43.064430 3209 raft_consensus.cc:1261] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 1 FOLLOWER]: Refusing update from remote peer 8e2ebe41f19d468d847ce357aec49ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:43.064430 3041 raft_consensus.cc:1261] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 1 FOLLOWER]: Refusing update from remote peer 8e2ebe41f19d468d847ce357aec49ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:43.065251 3495 consensus_queue.cc:1035] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2555ee4da3ce4c41a96db6087a378b28" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:43.065514 3497 consensus_queue.cc:1035] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eac6bff44d542bd9ea631f877c97e54" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:43.074836 3501 mvcc.cc:205] Tried to move back new op lower bound from 7315417428223442944 to 7315417428143697920. Current Snapshot: MvccSnapshot[applied={T|T < 7315417428223442944}]
I20260806 04:29:43.077772 3503 mvcc.cc:205] Tried to move back new op lower bound from 7315417428223442944 to 7315417428143697920. Current Snapshot: MvccSnapshot[applied={T|T < 7315417428223442944}]
I20260806 04:29:43.084167 2839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:43.084347 2839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:43.086963 2839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24da59fa51f543ca97538417b402473f: Sending DeleteTablet for 3 replicas of tablet f1cea9ce596b479fa4a996c2358b3545
I20260806 04:29:43.087983 3015 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cea9ce596b479fa4a996c2358b3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:43 UTC) from {username='slave'} at 127.0.0.1:42280
I20260806 04:29:43.088027 3185 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cea9ce596b479fa4a996c2358b3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:43 UTC) from {username='slave'} at 127.0.0.1:36478
I20260806 04:29:43.088542 3513 tablet_replica.cc:330] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: stopping tablet replica
I20260806 04:29:43.088567 3368 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cea9ce596b479fa4a996c2358b3545 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:43 UTC) from {username='slave'} at 127.0.0.1:43612
I20260806 04:29:43.088927 3513 raft_consensus.cc:2229] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:43.089150 3514 tablet_replica.cc:330] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: stopping tablet replica
I20260806 04:29:43.089355 3512 tablet_replica.cc:330] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54: stopping tablet replica
I20260806 04:29:43.089586 3513 raft_consensus.cc:2258] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:43.089702 3512 raft_consensus.cc:2229] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:43.089951 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2926
I20260806 04:29:43.090039 3512 raft_consensus.cc:2258] T f1cea9ce596b479fa4a996c2358b3545 P 6eac6bff44d542bd9ea631f877c97e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:43.101971 3513 ts_tablet_manager.cc:1905] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:43.106132 3513 ts_tablet_manager.cc:1918] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:43.106310 3513 log.cc:1196] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/f1cea9ce596b479fa4a996c2358b3545
I20260806 04:29:43.106770 3513 ts_tablet_manager.cc:1939] T f1cea9ce596b479fa4a996c2358b3545 P 2555ee4da3ce4c41a96db6087a378b28: Deleting consensus metadata
I20260806 04:29:43.108881 2819 catalog_manager.cc:4841] TS 2555ee4da3ce4c41a96db6087a378b28 (127.28.94.2:41559): tablet f1cea9ce596b479fa4a996c2358b3545 (table test-table [id=0e18035224bd43909712a427f6b7f178]) successfully deleted
I20260806 04:29:43.114588 3514 raft_consensus.cc:2229] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:43.115128 3514 raft_consensus.cc:2258] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:43.116962 3514 ts_tablet_manager.cc:1905] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:43.120155 3514 ts_tablet_manager.cc:1918] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:43.120287 3514 log.cc:1196] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/f1cea9ce596b479fa4a996c2358b3545
I20260806 04:29:43.120604 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3105
I20260806 04:29:43.120625 3514 ts_tablet_manager.cc:1939] T f1cea9ce596b479fa4a996c2358b3545 P 8e2ebe41f19d468d847ce357aec49ad8: Deleting consensus metadata
W20260806 04:29:43.120554 2814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:44865 (error 108)
I20260806 04:29:43.124043 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.123669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44865 (local address 127.0.0.1:42288)
0806 04:29:43.123833 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:43.123930 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:29:43.125043 2814 catalog_manager.cc:4584] TS 6eac6bff44d542bd9ea631f877c97e54 (127.28.94.1:44865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1cea9ce596b479fa4a996c2358b3545: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44865: connect: Connection refused (error 111)
I20260806 04:29:43.125326 2814 catalog_manager.cc:4841] TS 8e2ebe41f19d468d847ce357aec49ad8 (127.28.94.3:38747): tablet f1cea9ce596b479fa4a996c2358b3545 (table test-table [id=0e18035224bd43909712a427f6b7f178]) successfully deleted
I20260806 04:29:43.132364 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3281
I20260806 04:29:43.151702 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 2793
2026-08-06T04:29:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1232 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 04:29:43.186579 29048 mini_oidc.cc:152] Starting JWKS server
I20260806 04:29:43.188486 29048 webserver.cc:460] Webserver started at https://127.0.0.1:46165/ using document root <none> and password file <none>
I20260806 04:29:43.188687 29048 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:29:43.189021 29048 webserver.cc:460] Webserver started at http://127.0.0.1:37823/ using document root <none> and password file <none>
2026-08-06T04:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:43Z Disabled control of system clock
I20260806 04:29:43.200450 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34383
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:34383
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46165/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/testchainca.cert with env {}
W20260806 04:29:43.321918 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:43.322263 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:43.322306 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:43.326553 3524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:43.326655 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:43.326675 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:43.326700 3524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:43.326717 3524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:29:43.326733 3524 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:29:43.326749 3524 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46165/jwks.json
I20260806 04:29:43.332613 3524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:34383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46165/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:43.333863 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:43.334952 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:43.344138 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:43.344157 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:43.344152 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:43.344761 3524 server_base.cc:1034] running on GCE node
I20260806 04:29:43.345429 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:43.346738 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:43.347916 3524 hybrid_clock.cc:648] HybridClock initialized: now 1785990583347893 us; error 36 us; skew 500 ppm
I20260806 04:29:43.350051 3524 webserver.cc:460] Webserver started at http://127.28.94.62:45787/ using document root <none> and password file <none>
I20260806 04:29:43.350698 3524 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:43.350770 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:43.350982 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:43.353289 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "6d368cbd3a08474499414d17295276fe"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "b73bda72e4b2f51154c341ceebec7641"
server_key_iv: "1eaaa0a3fab8260f33c3e07222199dc6"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.353900 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "6d368cbd3a08474499414d17295276fe"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "b73bda72e4b2f51154c341ceebec7641"
server_key_iv: "1eaaa0a3fab8260f33c3e07222199dc6"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.357976 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:29:43.360651 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:43.362074 3524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:29:43.362244 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "6d368cbd3a08474499414d17295276fe"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "b73bda72e4b2f51154c341ceebec7641"
server_key_iv: "1eaaa0a3fab8260f33c3e07222199dc6"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.362365 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:43.378804 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:43.379616 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:43.379828 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:43.387257 3524 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:34383
I20260806 04:29:43.387331 3618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:34383 every 8 connection(s)
I20260806 04:29:43.388326 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:43.391984 3619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:43.398016 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3524
I20260806 04:29:43.398175 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:43.398476 29048 external_mini_cluster.cc:1428] Setting key 9d11f058ce98df3b7ee96be4c1c65c6b
I20260806 04:29:43.398862 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe: Bootstrap starting.
I20260806 04:29:43.401819 3619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:43.402695 3619 log.cc:824] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:43.404875 3619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe: No bootstrap required, opened a new log
I20260806 04:29:43.408509 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.400057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.28.94.62:34383)
0806 04:29:43.400866 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.400882 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.400966 (+ 84us) server_negotiation.cc:407] Connection header received
0806 04:29:43.401691 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.401710 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.402226 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.402577 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:43.403052 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.404060 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.405047 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.405547 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.405706 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.406566 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:43.406572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:43.406587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:43.407018 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:43.407081 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:43.407103 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:43.407195 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:43.407350 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 04:29:43.407670 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":168,"threads_started":1}
I20260806 04:29:43.408874 3619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } }
I20260806 04:29:43.409173 3619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:43.409235 3619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d368cbd3a08474499414d17295276fe, State: Initialized, Role: FOLLOWER
I20260806 04:29:43.409696 3619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } }
I20260806 04:29:43.409837 3619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:43.409905 3619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:43.410007 3619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:43.411149 3619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } }
I20260806 04:29:43.411506 3619 leader_election.cc:302] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d368cbd3a08474499414d17295276fe; no voters:
I20260806 04:29:43.411772 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:29:43.411917 3619 leader_election.cc:288] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:43.412006 3625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:43.412233 3625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [term 1 LEADER]: Becoming Leader. State: Replica: 6d368cbd3a08474499414d17295276fe, State: Running, Role: LEADER
I20260806 04:29:43.412606 3625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } }
I20260806 04:29:43.413385 3619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:43.414526 3626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d368cbd3a08474499414d17295276fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } } }
I20260806 04:29:43.414678 3626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:43.415089 3638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:43.415076 3628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d368cbd3a08474499414d17295276fe. Latest consensus state: current_term: 1 leader_uuid: "6d368cbd3a08474499414d17295276fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d368cbd3a08474499414d17295276fe" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34383 } } }
I20260806 04:29:43.415982 3628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:43.418497 3638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:43.425166 3638 catalog_manager.cc:1347] Generated new cluster ID: 04b656ad163146928a2ea3543b38f7d1
I20260806 04:29:43.425266 3638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:43.432505 3638 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:43.433677 3638 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:43.438692 3638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d368cbd3a08474499414d17295276fe: Generated new TSK 0
I20260806 04:29:43.439453 3638 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:29:43.481434 3665 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 04:29:43.481619 3665 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:29:43.482136 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.457083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.28.94.62:34383)
0806 04:29:43.457221 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.457225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.457541 (+ 316us) server_negotiation.cc:407] Connection header received
0806 04:29:43.457805 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.457808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.457863 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.457955 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:29:43.458242 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.458833 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.460064 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.460291 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.460363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.460774 (+ 411us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:29:43.481615 (+ 20841us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:29:43.481958 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 04:29:43.482280 3622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 04:29:43.483493 3664 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:43.456992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:34383 (local address 127.0.0.1:52862)
0806 04:29:43.457453 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:43.457468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:43.457627 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:43.457987 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:43.457990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:43.457998 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:29:43.458132 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:43.458138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.458932 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.458936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:43.459953 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:43.459964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.460399 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.460402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:43.460500 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.460568 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:29:43.481898 (+ 21330us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:29:43.482222 (+ 324us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.94.62:34383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.94.62:34383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 04:29:43.484347 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3524
2026-08-06T04:29:43Z chronyd exiting
I20260806 04:29:43.532505 29048 test_util.cc:175] -----------------------------------------------
I20260806 04:29:43.532605 29048 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785990554567992-29048-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 04:29:43.533851 29048 mini_oidc.cc:152] Starting JWKS server
I20260806 04:29:43.534777 29048 webserver.cc:460] Webserver started at https://127.0.0.1:35531/ using document root <none> and password file <none>
I20260806 04:29:43.534870 29048 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:29:43.535094 29048 webserver.cc:460] Webserver started at http://127.0.0.1:38635/ using document root <none> and password file <none>
2026-08-06T04:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:43Z Disabled control of system clock
I20260806 04:29:43.546056 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45567
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:45567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/testchainca.cert with env {}
W20260806 04:29:43.667717 3674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:43.668048 3674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:43.668092 3674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:43.672345 3674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:43.672425 3674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:43.672439 3674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:43.672456 3674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:43.672472 3674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:29:43.672487 3674 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:29:43.672499 3674 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35531/jwks.json
I20260806 04:29:43.678534 3674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:45567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:43.679767 3674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:43.680846 3674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:43.690182 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:43.690172 3685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:43.690181 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:43.690286 3674 server_base.cc:1034] running on GCE node
I20260806 04:29:43.691128 3674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:43.692396 3674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:43.693580 3674 hybrid_clock.cc:648] HybridClock initialized: now 1785990583693561 us; error 54 us; skew 500 ppm
I20260806 04:29:43.695708 3674 webserver.cc:460] Webserver started at http://127.28.94.62:40067/ using document root <none> and password file <none>
I20260806 04:29:43.696364 3674 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:43.696439 3674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:43.696661 3674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:43.698973 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "0f82282402ae4bad8bf1b573b69a6cff"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "1c2b792c18f20f327dee74ad7bdb1997"
server_key_iv: "84c307c3bbceff17f13681c6c2660cc2"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.699630 3674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "0f82282402ae4bad8bf1b573b69a6cff"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "1c2b792c18f20f327dee74ad7bdb1997"
server_key_iv: "84c307c3bbceff17f13681c6c2660cc2"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.703992 3674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:29:43.706912 3692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:43.708343 3674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:43.708580 3674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "0f82282402ae4bad8bf1b573b69a6cff"
format_stamp: "Formatted at 2026-08-06 04:29:43 on dist-test-slave-ncp9"
server_key: "1c2b792c18f20f327dee74ad7bdb1997"
server_key_iv: "84c307c3bbceff17f13681c6c2660cc2"
server_key_version: "encryptionkey@0"
I20260806 04:29:43.708724 3674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:43.724761 3674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:43.725682 3674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:43.725895 3674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:43.733217 3674 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:45567
I20260806 04:29:43.733251 3764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:45567 every 8 connection(s)
I20260806 04:29:43.734347 3674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:43.738209 3765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:43.743659 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3674
I20260806 04:29:43.743782 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:43.744051 29048 external_mini_cluster.cc:1428] Setting key 3601530632d8251857c45e8751f133bd
I20260806 04:29:43.745056 3765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: Bootstrap starting.
I20260806 04:29:43.748250 3765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:43.749378 3765 log.cc:824] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:43.751888 3765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: No bootstrap required, opened a new log
I20260806 04:29:43.755601 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.745669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.28.94.62:45567)
0806 04:29:43.746488 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.746511 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.746581 (+ 70us) server_negotiation.cc:407] Connection header received
0806 04:29:43.747530 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.747558 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.748141 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.748590 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:43.749153 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.750446 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.751462 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.752186 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.752371 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.753221 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:43.753228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:43.753247 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:43.753800 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:43.753883 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:43.753911 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:43.754015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:43.754179 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 04:29:43.754496 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":137,"threads_started":1}
I20260806 04:29:43.756702 3765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } }
I20260806 04:29:43.756990 3765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:43.757071 3765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f82282402ae4bad8bf1b573b69a6cff, State: Initialized, Role: FOLLOWER
I20260806 04:29:43.757529 3765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } }
I20260806 04:29:43.757666 3765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:43.757731 3765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:43.757826 3765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:43.758924 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:29:43.758953 3765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } }
I20260806 04:29:43.759359 3765 leader_election.cc:302] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f82282402ae4bad8bf1b573b69a6cff; no voters:
I20260806 04:29:43.759802 3765 leader_election.cc:288] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:43.759913 3770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:43.760196 3770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [term 1 LEADER]: Becoming Leader. State: Replica: 0f82282402ae4bad8bf1b573b69a6cff, State: Running, Role: LEADER
I20260806 04:29:43.760552 3770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } }
I20260806 04:29:43.761317 3765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:43.762310 3771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f82282402ae4bad8bf1b573b69a6cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } } }
I20260806 04:29:43.762465 3771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:43.762916 3772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f82282402ae4bad8bf1b573b69a6cff. Latest consensus state: current_term: 1 leader_uuid: "0f82282402ae4bad8bf1b573b69a6cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f82282402ae4bad8bf1b573b69a6cff" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 45567 } } }
I20260806 04:29:43.763095 3772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff [sys.catalog]: This master's current role is: LEADER
W20260806 04:29:43.766364 3787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:29:43.766446 3787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:29:43.766530 3780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:43.767287 3780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:43.773638 3780 catalog_manager.cc:1347] Generated new cluster ID: 8842d135f2724054b50bd0a373ca2a34
I20260806 04:29:43.773725 3780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:43.780836 3780 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:43.781893 3780 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:43.786762 3780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f82282402ae4bad8bf1b573b69a6cff: Generated new TSK 0
I20260806 04:29:43.787566 3780 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:29:43.827584 3804 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 04:29:43.827690 3804 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:29:43.828383 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:43.804652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.28.94.62:45567)
0806 04:29:43.804800 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:43.804804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:43.804923 (+ 119us) server_negotiation.cc:407] Connection header received
0806 04:29:43.805159 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:43.805162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:43.805220 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:43.805321 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:29:43.805596 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.806171 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.807425 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.807647 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.807720 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.808083 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:29:43.827976 (+ 19893us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:29:43.828254 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 04:29:43.828454 3768 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 04:29:43.828630 3802 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:43.804460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:45567 (local address 127.0.0.1:36668)
0806 04:29:43.804817 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:43.804833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:43.805014 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:43.805339 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:43.805342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:43.805351 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:29:43.805481 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:43.805488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.806284 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.806288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:43.807309 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:43.807322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:43.807769 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:43.807773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:43.807867 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:43.807902 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:29:43.828218 (+ 20316us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:29:43.828441 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.94.62:45567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260806 04:29:43.829313 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3674
2026-08-06T04:29:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (346 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 04:29:43.880147 29048 mini_oidc.cc:152] Starting JWKS server
I20260806 04:29:43.881132 29048 webserver.cc:460] Webserver started at https://127.0.0.1:39697/ using document root <none> and password file <none>
I20260806 04:29:43.881248 29048 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:29:43.881515 29048 webserver.cc:460] Webserver started at http://127.0.0.1:44407/ using document root <none> and password file <none>
2026-08-06T04:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:43Z Disabled control of system clock
I20260806 04:29:43.892822 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34375
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:34375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:29:44.016275 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:44.016605 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:44.016657 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:44.021015 3810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:44.021090 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:44.021103 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:44.021121 3810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:44.021137 3810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:29:44.021166 3810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:29:44.021193 3810 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39697/jwks.json
I20260806 04:29:44.026504 3810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:34375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:44.027812 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:44.029492 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:44.039052 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:44.039057 3820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:44.039057 3817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:44.039462 3810 server_base.cc:1034] running on GCE node
I20260806 04:29:44.040148 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:44.041530 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:44.042699 3810 hybrid_clock.cc:648] HybridClock initialized: now 1785990584042682 us; error 35 us; skew 500 ppm
I20260806 04:29:44.044857 3810 webserver.cc:460] Webserver started at http://127.28.94.62:36477/ using document root <none> and password file <none>
I20260806 04:29:44.045565 3810 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:44.045655 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:44.045905 3810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:44.048348 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "932962ba7f2f46ed93ae3678bd2369d8"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "c0e3803edf5a2b18cb3742d6e3b9a1d8"
server_key_iv: "5834f6760933233545cfcedc47efa437"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.049044 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "932962ba7f2f46ed93ae3678bd2369d8"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "c0e3803edf5a2b18cb3742d6e3b9a1d8"
server_key_iv: "5834f6760933233545cfcedc47efa437"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.053395 3810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:29:44.056279 3827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:44.057770 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:44.057920 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "932962ba7f2f46ed93ae3678bd2369d8"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "c0e3803edf5a2b18cb3742d6e3b9a1d8"
server_key_iv: "5834f6760933233545cfcedc47efa437"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.058059 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:44.078159 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:44.079005 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:44.079213 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:44.086998 3810 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:34375
I20260806 04:29:44.087016 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:34375 every 8 connection(s)
I20260806 04:29:44.088189 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:44.090569 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3810
I20260806 04:29:44.090689 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:44.090956 29048 external_mini_cluster.cc:1428] Setting key eac9aa14f5700132e11d68fcc9938bf2
I20260806 04:29:44.092425 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:44.099560 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8: Bootstrap starting.
I20260806 04:29:44.100968 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:44.092414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.28.94.62:34375)
0806 04:29:44.093165 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:44.093183 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:44.093225 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:29:44.093910 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:44.093930 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:44.094376 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:44.094701 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:44.095106 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.096123 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.097376 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.097894 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.098018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.098981 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:44.098987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:44.099004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:44.099431 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:44.099494 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:44.099517 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:44.099595 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:44.099811 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 04:29:44.100124 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260806 04:29:44.102388 3902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:44.103358 3902 log.cc:824] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:44.104331 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:29:44.105861 3902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8: No bootstrap required, opened a new log
I20260806 04:29:44.109782 3902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } }
I20260806 04:29:44.110064 3902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:44.110101 3902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932962ba7f2f46ed93ae3678bd2369d8, State: Initialized, Role: FOLLOWER
I20260806 04:29:44.110544 3902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } }
I20260806 04:29:44.110719 3902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:44.110795 3902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:44.110911 3902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:44.112077 3902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } }
I20260806 04:29:44.112452 3902 leader_election.cc:302] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932962ba7f2f46ed93ae3678bd2369d8; no voters:
I20260806 04:29:44.112821 3902 leader_election.cc:288] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:44.112972 3907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:44.113224 3907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [term 1 LEADER]: Becoming Leader. State: Replica: 932962ba7f2f46ed93ae3678bd2369d8, State: Running, Role: LEADER
I20260806 04:29:44.113561 3907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } }
I20260806 04:29:44.114281 3902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:44.115552 3909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932962ba7f2f46ed93ae3678bd2369d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } } }
I20260806 04:29:44.115552 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932962ba7f2f46ed93ae3678bd2369d8. Latest consensus state: current_term: 1 leader_uuid: "932962ba7f2f46ed93ae3678bd2369d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932962ba7f2f46ed93ae3678bd2369d8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 34375 } } }
I20260806 04:29:44.115728 3909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:44.115728 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:44.116199 3920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:44.119297 3920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:44.126026 3920 catalog_manager.cc:1347] Generated new cluster ID: 98c1454db0464698a2d14ed07d305db7
I20260806 04:29:44.126142 3920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:44.157553 3920 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:44.158995 3920 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:44.167788 3920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932962ba7f2f46ed93ae3678bd2369d8: Generated new TSK 0
I20260806 04:29:44.168820 3920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:29:44.185593 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:44.179776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.28.94.62:34375)
0806 04:29:44.179946 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:44.179951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:44.180075 (+ 124us) server_negotiation.cc:407] Connection header received
0806 04:29:44.180258 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:44.180261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:44.180328 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:44.180466 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:29:44.180753 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.181623 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.182629 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.183000 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.183116 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.183446 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:29:44.185170 (+ 1724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:29:44.185437 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":86}
W20260806 04:29:44.185683 3905 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 04:29:44.187191 3939 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:44.179626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:34375 (local address 127.0.0.1:34402)
0806 04:29:44.179968 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:44.179984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:44.180109 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:44.180507 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:44.180510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:44.180518 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:29:44.180637 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:44.180642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.181777 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.181780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:44.182497 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:44.182509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.183140 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.183143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:44.183217 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.183248 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:29:44.185352 (+ 2104us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:29:44.186998 (+ 1646us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.94.62:34375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1421,"thread_start_us":117,"threads_started":1}
I20260806 04:29:44.188030 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3810
E20260806 04:29:44.198717 3940 webserver.cc:533] Webserver: sslize failed:
E20260806 04:29:44.198824 3940 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T04:29:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (354 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T04:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:44Z Disabled control of system clock
I20260806 04:29:44.245688 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39229
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39229
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/testchainca.cert with env {}
W20260806 04:29:44.366703 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:44.367043 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:44.367105 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:44.371430 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:44.371529 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:44.371551 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:44.371572 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:44.371591 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:29:44.371621 3945 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:29:44.371647 3945 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 04:29:44.377655 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:44.378962 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:44.380151 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:44.389664 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:44.389664 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:44.389783 3945 server_base.cc:1034] running on GCE node
W20260806 04:29:44.389694 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:44.390587 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:44.391875 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:44.393039 3945 hybrid_clock.cc:648] HybridClock initialized: now 1785990584393007 us; error 46 us; skew 500 ppm
I20260806 04:29:44.398236 3945 webserver.cc:460] Webserver started at http://127.28.94.62:42187/ using document root <none> and password file <none>
I20260806 04:29:44.398957 3945 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:44.399041 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:44.399263 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:44.401682 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "bc31ba3110f84459a30eec88684dc008"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "770480547e0e1ca552bdf035fca24a21"
server_key_iv: "24f9e5c8ff1fc7633c08d1c9464fc912"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.402418 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "bc31ba3110f84459a30eec88684dc008"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "770480547e0e1ca552bdf035fca24a21"
server_key_iv: "24f9e5c8ff1fc7633c08d1c9464fc912"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.406777 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:44.409868 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:44.411326 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:29:44.411489 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "bc31ba3110f84459a30eec88684dc008"
format_stamp: "Formatted at 2026-08-06 04:29:44 on dist-test-slave-ncp9"
server_key: "770480547e0e1ca552bdf035fca24a21"
server_key_iv: "24f9e5c8ff1fc7633c08d1c9464fc912"
server_key_version: "encryptionkey@0"
I20260806 04:29:44.411630 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:44.450779 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:44.451651 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:44.451857 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:44.459386 3945 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39229
I20260806 04:29:44.459415 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39229 every 8 connection(s)
I20260806 04:29:44.460467 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:44.463486 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 3945
I20260806 04:29:44.463614 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:44.463901 29048 external_mini_cluster.cc:1428] Setting key 5d2eaa7e5424368f7897da1fd688600b
I20260806 04:29:44.464097 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:44.472461 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: Bootstrap starting.
I20260806 04:29:44.475255 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:44.465447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.28.94.62:39229)
0806 04:29:44.466252 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:44.466273 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:44.466323 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:29:44.467200 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:44.467230 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:44.467825 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:44.468261 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:44.468773 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.470262 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.471241 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.471965 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.472138 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.472991 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:44.472997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:44.473017 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:29:44.473544 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:44.473635 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:29:44.473665 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:44.473767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:44.473907 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 04:29:44.474223 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":164,"threads_started":1}
I20260806 04:29:44.475612 4042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:44.476425 4042 log.cc:824] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:44.478473 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:29:44.478816 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: No bootstrap required, opened a new log
I20260806 04:29:44.482851 4042 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } }
I20260806 04:29:44.483160 4042 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:44.483229 4042 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc31ba3110f84459a30eec88684dc008, State: Initialized, Role: FOLLOWER
I20260806 04:29:44.483724 4042 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } }
I20260806 04:29:44.483871 4042 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:44.483944 4042 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:44.484055 4042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:44.485287 4042 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } }
I20260806 04:29:44.485730 4042 leader_election.cc:302] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc31ba3110f84459a30eec88684dc008; no voters:
I20260806 04:29:44.486135 4042 leader_election.cc:288] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:44.486334 4048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:44.486580 4048 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [term 1 LEADER]: Becoming Leader. State: Replica: bc31ba3110f84459a30eec88684dc008, State: Running, Role: LEADER
I20260806 04:29:44.486943 4048 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } }
I20260806 04:29:44.487720 4042 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:44.489053 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc31ba3110f84459a30eec88684dc008. Latest consensus state: current_term: 1 leader_uuid: "bc31ba3110f84459a30eec88684dc008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } } }
I20260806 04:29:44.489204 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:44.489955 4050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc31ba3110f84459a30eec88684dc008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31ba3110f84459a30eec88684dc008" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39229 } } }
I20260806 04:29:44.491360 4050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008 [sys.catalog]: This master's current role is: LEADER
W20260806 04:29:44.492795 4070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:29:44.492874 4070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:29:44.493327 4059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:44.494076 4059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:44.501444 4059 catalog_manager.cc:1347] Generated new cluster ID: b7c073cf21a14bfcb24c8c13600c89f5
I20260806 04:29:44.501538 4059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:44.521487 4059 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:44.522724 4059 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:44.530910 4059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc31ba3110f84459a30eec88684dc008: Generated new TSK 0
I20260806 04:29:44.531894 4059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:29:44.589898 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:44.546237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.28.94.62:39229)
0806 04:29:44.546382 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:44.546388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:44.549322 (+ 2934us) server_negotiation.cc:407] Connection header received
0806 04:29:44.549519 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:44.549522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:44.549587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:44.549701 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:29:44.549976 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.550539 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.551843 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.552087 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.552174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.552546 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:29:44.589445 (+ 36899us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:29:44.589765 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 04:29:44.589985 4046 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39438: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 04:29:44.590265 4081 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:44.546132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39229 (local address 127.0.0.1:39438)
0806 04:29:44.549183 (+ 3051us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:44.549203 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:44.549366 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:44.549709 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:44.549712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:44.549720 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:29:44.549857 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:44.549865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.550652 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.550656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:44.551672 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:44.551687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:44.552221 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:44.552225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:44.552329 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:44.552365 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:29:44.589756 (+ 37391us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:29:44.589974 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.94.62:39229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":2924,"thread_start_us":121,"threads_started":1}
I20260806 04:29:44.590975 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 3945
2026-08-06T04:29:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (386 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (140 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (138 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:44 dist-test-slave-ncp9 krb5kdc[4096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:44 dist-test-slave-ncp9 krb5kdc[4096](info): set up 2 sockets
Aug 06 04:29:44 dist-test-slave-ncp9 krb5kdc[4096](info): commencing operation
krb5kdc: starting...
W20260806 04:29:46.006697 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:29:46.142459 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37071
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:46.266433 4111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:46.266772 4111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:46.266834 4111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:46.271137 4111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:46.271234 4111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:46.271265 4111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:46.271286 4111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:46.271304 4111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:46.277168 4111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:46.278434 4111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:46.279585 4111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:46.289052 4123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:46.289043 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:46.289036 4119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:46.289222 4111 server_base.cc:1034] running on GCE node
I20260806 04:29:46.290030 4111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:46.291313 4111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:46.292490 4111 hybrid_clock.cc:648] HybridClock initialized: now 1785990586292466 us; error 35 us; skew 500 ppm
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:46.299072 4111 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:29:46.300273 4111 webserver.cc:460] Webserver started at http://127.28.94.62:42117/ using document root <none> and password file <none>
I20260806 04:29:46.300976 4111 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:46.301065 4111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:46.301306 4111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:46.303619 4111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5a9c30145e4247b29c9174da79ec68bb"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "7b7781b621c3e11f9757dc760632d27e"
server_key_iv: "edcd7e32c75f31ab1bea30c09c513b2a"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.304261 4111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5a9c30145e4247b29c9174da79ec68bb"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "7b7781b621c3e11f9757dc760632d27e"
server_key_iv: "edcd7e32c75f31ab1bea30c09c513b2a"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.308549 4111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:46.311398 4131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.312819 4111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:29:46.313033 4111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5a9c30145e4247b29c9174da79ec68bb"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "7b7781b621c3e11f9757dc760632d27e"
server_key_iv: "edcd7e32c75f31ab1bea30c09c513b2a"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.313161 4111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:46.339483 4111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:46.342834 4111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:46.343050 4111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:46.350863 4111 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:37071
I20260806 04:29:46.350883 4210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:37071 every 8 connection(s)
I20260806 04:29:46.351984 4111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:46.355142 4211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:46.361172 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4111
I20260806 04:29:46.361325 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:46.361621 29048 external_mini_cluster.cc:1428] Setting key 515dab9c0be9cb35bd7df65c2c18f854
I20260806 04:29:46.361755 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb: Bootstrap starting.
I20260806 04:29:46.364576 4211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:46.365585 4211 log.cc:824] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:46.368094 4211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb: No bootstrap required, opened a new log
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:46.372922 4211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } }
I20260806 04:29:46.373231 4211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:46.373306 4211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9c30145e4247b29c9174da79ec68bb, State: Initialized, Role: FOLLOWER
I20260806 04:29:46.373814 4211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } }
I20260806 04:29:46.373961 4211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:46.374032 4211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:46.374140 4211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:46.375126 4211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } }
I20260806 04:29:46.375510 4211 leader_election.cc:302] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a9c30145e4247b29c9174da79ec68bb; no voters:
I20260806 04:29:46.375880 4211 leader_election.cc:288] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:46.376029 4217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:46.376251 4217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [term 1 LEADER]: Becoming Leader. State: Replica: 5a9c30145e4247b29c9174da79ec68bb, State: Running, Role: LEADER
I20260806 04:29:46.376585 4217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } }
I20260806 04:29:46.377419 4211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:46.377612 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:46.363220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.28.94.62:37071)
0806 04:29:46.363706 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:46.363723 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:46.363763 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:29:46.364461 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:46.364481 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:46.364886 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:46.365242 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:46.366199 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.367334 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.368495 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.369066 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.369203 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:46.372225 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:46.372231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:46.372250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:46.372287 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:46.374132 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.374605 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.374612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.374619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.374754 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.375045 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.375049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.375051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.375550 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:46.375750 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:46.376088 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 04:29:46.376576 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260806 04:29:46.377983 4219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a9c30145e4247b29c9174da79ec68bb. Latest consensus state: current_term: 1 leader_uuid: "5a9c30145e4247b29c9174da79ec68bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } } }
I20260806 04:29:46.378119 4219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:46.378539 4224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:46.378660 4218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a9c30145e4247b29c9174da79ec68bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9c30145e4247b29c9174da79ec68bb" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37071 } } }
I20260806 04:29:46.378816 4218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:46.382171 4224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:46.388209 4224 catalog_manager.cc:1347] Generated new cluster ID: 536221160d4b49d9b7ac91a2d7cb9c73
I20260806 04:29:46.388311 4224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:46.406903 4224 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:46.408376 4224 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:46.414232 4224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a9c30145e4247b29c9174da79ec68bb: Generated new TSK 0
I20260806 04:29:46.415027 4224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:29:46.453945 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:46.578511 4243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:46.578878 4243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:46.578935 4243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:46.583431 4243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:46.583528 4243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:46.583592 4243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:46.589485 4243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:46.590737 4243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:46.591902 4243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:46.602393 4252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:46.602396 4255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:46.602689 4243 server_base.cc:1034] running on GCE node
W20260806 04:29:46.602393 4253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:46.603257 4243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:46.604017 4243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:46.605180 4243 hybrid_clock.cc:648] HybridClock initialized: now 1785990586605154 us; error 35 us; skew 500 ppm
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:46.608062 4243 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:29:46.609138 4243 webserver.cc:460] Webserver started at http://127.28.94.1:41663/ using document root <none> and password file <none>
I20260806 04:29:46.609776 4243 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:46.609854 4243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:46.610097 4243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:46.612430 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "1805d65583094d4598b565df68364706"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "ff5658a4007509ff834dded6c808b16c"
server_key_iv: "ba982c32c7cba6b48066e4eff3cd1907"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.613095 4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "1805d65583094d4598b565df68364706"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "ff5658a4007509ff834dded6c808b16c"
server_key_iv: "ba982c32c7cba6b48066e4eff3cd1907"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.617272 4243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:46.620038 4262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.621500 4243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:46.621671 4243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "1805d65583094d4598b565df68364706"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "ff5658a4007509ff834dded6c808b16c"
server_key_iv: "ba982c32c7cba6b48066e4eff3cd1907"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.621797 4243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:46.635977 4243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:46.639159 4243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:46.639369 4243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:46.640053 4243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:46.641055 4243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:46.641108 4243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.641155 4243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:46.641185 4243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.650489 4243 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:45037
I20260806 04:29:46.650508 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:45037 every 8 connection(s)
I20260806 04:29:46.651549 4243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:46.652307 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4243
I20260806 04:29:46.652431 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:46.652729 29048 external_mini_cluster.cc:1428] Setting key d57c728e2a5f23d5a967f4fce2229b46
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:29:46.666905 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:46.653611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:41929 (local address 127.28.94.62:37071)
0806 04:29:46.653751 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:46.653756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:46.654889 (+ 1133us) server_negotiation.cc:407] Connection header received
0806 04:29:46.656372 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:46.656375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:46.656438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:46.656536 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:46.657857 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.658619 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.659900 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.660283 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.660408 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:46.663103 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:46.663109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:46.663111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:46.663154 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:46.664567 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.665195 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.665198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.665200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.665286 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.665649 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.665651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.665652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.665840 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:46.665911 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:46.666607 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 04:29:46.666760 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:29:46.668110 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:46.654122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37071 (local address 127.28.94.1:41929)
0806 04:29:46.654704 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:46.654765 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:46.656135 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:46.656698 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:46.656706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:46.657151 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:46.657688 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:46.657703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.658746 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.658749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:46.659758 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:46.659770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.660443 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.660446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:46.660613 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:46.661268 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:46.661287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:46.662870 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:46.664715 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:46.664722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:46.664740 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:46.665100 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:46.665355 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:46.665358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:46.665360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:46.665572 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:46.665947 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:46.665955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:46.666307 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 04:29:46.666535 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 04:29:46.666959 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260806 04:29:46.669574 4431 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37071
I20260806 04:29:46.669907 4431 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:46.670491 4431 heartbeater.cc:511] Master 127.28.94.62:37071 requested a full tablet report, sending...
I20260806 04:29:46.672290 4157 ts_manager.cc:194] Registered new tserver with Master: 1805d65583094d4598b565df68364706 (127.28.94.1:45037)
I20260806 04:29:46.673542 4157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:41929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:29:46.681797 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:46.805204 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:46.805531 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:46.805577 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:46.809930 4436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:46.810016 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:46.810081 4436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:46.815943 4436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:46.817245 4436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:46.818374 4436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:46.829133 4450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:46.829133 4448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:46.829478 4436 server_base.cc:1034] running on GCE node
W20260806 04:29:46.829139 4447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:46.830035 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:46.830802 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:46.831970 4436 hybrid_clock.cc:648] HybridClock initialized: now 1785990586831950 us; error 35 us; skew 500 ppm
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:46.834951 4436 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:29:46.836053 4436 webserver.cc:460] Webserver started at http://127.28.94.2:39085/ using document root <none> and password file <none>
I20260806 04:29:46.836683 4436 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:46.836756 4436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:46.837033 4436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:46.839306 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "067b73278ee44e22ba24fa0ea73e9541"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "53dfe93192761d6db22a01f3fca922e3"
server_key_iv: "86fc402a64ba431bb8140e01b444553f"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.839991 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "067b73278ee44e22ba24fa0ea73e9541"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "53dfe93192761d6db22a01f3fca922e3"
server_key_iv: "86fc402a64ba431bb8140e01b444553f"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.844235 4436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:29:46.847106 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.848480 4436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:46.848618 4436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "067b73278ee44e22ba24fa0ea73e9541"
format_stamp: "Formatted at 2026-08-06 04:29:46 on dist-test-slave-ncp9"
server_key: "53dfe93192761d6db22a01f3fca922e3"
server_key_iv: "86fc402a64ba431bb8140e01b444553f"
server_key_version: "encryptionkey@0"
I20260806 04:29:46.848753 4436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:46.874938 4436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:46.878698 4436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:46.878899 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:46.879550 4436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:46.880508 4436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:46.880568 4436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.880626 4436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:46.880653 4436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:46.889943 4436 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:34219
I20260806 04:29:46.889967 4610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:34219 every 8 connection(s)
I20260806 04:29:46.890992 4436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:46.900624 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4436
I20260806 04:29:46.900744 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:46.901077 29048 external_mini_cluster.cc:1428] Setting key 79f5c31bb85c374798002bd9d68308c9
Aug 06 04:29:46 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990586, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:46.906698 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:46.893054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:51723 (local address 127.28.94.62:37071)
0806 04:29:46.893203 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:46.893207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:46.894289 (+ 1082us) server_negotiation.cc:407] Connection header received
0806 04:29:46.895827 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:46.895831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:46.895889 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:46.895990 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:46.897430 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.897979 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.899252 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.899491 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.899593 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:46.902601 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:46.902605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:46.902607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:46.902639 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:46.904136 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.904814 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.904817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.904818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.904934 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:46.905365 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:46.905369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:46.905371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:46.905620 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:46.905723 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:46.906376 (+ 653us) server_negotiation.cc:299] Negotiation successful
0806 04:29:46.906524 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:29:46.907596 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:46.893543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37071 (local address 127.28.94.2:51723)
0806 04:29:46.894110 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:46.894162 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:46.895519 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:46.896161 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:46.896168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:46.896578 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:46.897253 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:46.897270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.898131 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.898134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:46.899100 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:46.899112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:46.899657 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:46.899660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:46.899825 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:46.900518 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:46.900537 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:46.902410 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:46.904281 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:46.904289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:46.904311 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:46.904713 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:46.905034 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:46.905038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:46.905040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:46.905266 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:46.905762 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:46.905770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:46.906095 (+ 325us) client_negotiation.cc:769] Sending connection context
0806 04:29:46.906277 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 04:29:46.906589 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":134,"threads_started":1}
I20260806 04:29:46.909209 4612 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37071
I20260806 04:29:46.909487 4612 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:46.910094 4612 heartbeater.cc:511] Master 127.28.94.62:37071 requested a full tablet report, sending...
I20260806 04:29:46.911398 4157 ts_manager.cc:194] Registered new tserver with Master: 067b73278ee44e22ba24fa0ea73e9541 (127.28.94.2:34219)
I20260806 04:29:46.912242 4157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:51723
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:29:46.931945 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:47.055984 4619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:47.056308 4619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:47.056353 4619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:47.060540 4619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:47.060629 4619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:47.060694 4619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:47.066449 4619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:47.067682 4619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:47.068789 4619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:47.079358 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:47.079370 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:47.079358 4628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:47.079588 4619 server_base.cc:1034] running on GCE node
I20260806 04:29:47.080235 4619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:47.081027 4619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:47.082214 4619 hybrid_clock.cc:648] HybridClock initialized: now 1785990587082184 us; error 33 us; skew 500 ppm
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.085211 4619 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:29:47.086283 4619 webserver.cc:460] Webserver started at http://127.28.94.3:41329/ using document root <none> and password file <none>
I20260806 04:29:47.086925 4619 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:47.087004 4619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:47.087247 4619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:47.089679 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "6b2c39707eee4e078c502eecf0ec03ea"
format_stamp: "Formatted at 2026-08-06 04:29:47 on dist-test-slave-ncp9"
server_key: "3cb3dbd2b55f18c0163f1ae6dde64665"
server_key_iv: "71f16ba086edd3709e773fc90bccc6e0"
server_key_version: "encryptionkey@0"
I20260806 04:29:47.090292 4619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "6b2c39707eee4e078c502eecf0ec03ea"
format_stamp: "Formatted at 2026-08-06 04:29:47 on dist-test-slave-ncp9"
server_key: "3cb3dbd2b55f18c0163f1ae6dde64665"
server_key_iv: "71f16ba086edd3709e773fc90bccc6e0"
server_key_version: "encryptionkey@0"
I20260806 04:29:47.094516 4619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:47.097424 4636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:47.098809 4619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:47.098942 4619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "6b2c39707eee4e078c502eecf0ec03ea"
format_stamp: "Formatted at 2026-08-06 04:29:47 on dist-test-slave-ncp9"
server_key: "3cb3dbd2b55f18c0163f1ae6dde64665"
server_key_iv: "71f16ba086edd3709e773fc90bccc6e0"
server_key_version: "encryptionkey@0"
I20260806 04:29:47.099063 4619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:47.109906 4619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:47.113269 4619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:47.113478 4619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:47.114137 4619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:47.115073 4619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:47.115123 4619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:47.115180 4619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:47.115209 4619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:47.124460 4619 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:46169
I20260806 04:29:47.124486 4783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:46169 every 8 connection(s)
I20260806 04:29:47.125530 4619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:47.130162 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4619
I20260806 04:29:47.130306 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:47.130589 29048 external_mini_cluster.cc:1428] Setting key 1699f1f89f7532ea3c1530ccf7cc6c4f
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:29:47.140228 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:47.127612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37441 (local address 127.28.94.62:37071)
0806 04:29:47.127752 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.127756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.128637 (+ 881us) server_negotiation.cc:407] Connection header received
0806 04:29:47.130119 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.130123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.130190 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.130291 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.131900 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.132435 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.133770 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.133993 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.134070 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:47.136637 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:47.136640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:47.136642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:47.136691 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:47.138111 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.138714 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.138717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.138719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.138819 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.139167 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.139170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.139171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.139377 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:47.139458 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:47.139978 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 04:29:47.140096 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:29:47.141227 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:47.127952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:37071 (local address 127.28.94.3:37441)
0806 04:29:47.128458 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:47.128507 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:47.129876 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:47.130539 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:47.130547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:47.131003 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:47.131738 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:47.131760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.132576 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.132580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:47.133642 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:47.133654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.134116 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.134119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:47.134263 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:47.134905 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:47.134924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:47.136425 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:47.138231 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:47.138237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:47.138254 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:47.138564 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:47.138917 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:47.138920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:47.138921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:47.139078 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:47.139455 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:47.139461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:47.139770 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 04:29:47.139940 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 04:29:47.140231 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260806 04:29:47.142591 4784 heartbeater.cc:347] Connected to a master server at 127.28.94.62:37071
I20260806 04:29:47.142860 4784 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:47.143424 4784 heartbeater.cc:511] Master 127.28.94.62:37071 requested a full tablet report, sending...
I20260806 04:29:47.144680 4157 ts_manager.cc:194] Registered new tserver with Master: 6b2c39707eee4e078c502eecf0ec03ea (127.28.94.3:46169)
I20260806 04:29:47.145334 4157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:37441
I20260806 04:29:47.157315 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/test-admin.keytab.
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.173292 29048 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 04:29:47.176550 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:47.174835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.28.94.1:45037)
0806 04:29:47.175139 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.175147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.175174 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:29:47.175264 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.175270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.175481 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.175625 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.176323 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53336: BlockingRecv error: recv got EOF from 127.0.0.1:53336 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260806 04:29:47.176730 29048 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.178170 29048 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.179442 29048 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:47.190428 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:47.180686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.28.94.1:45037)
0806 04:29:47.180829 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.180834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.180907 (+ 73us) server_negotiation.cc:407] Connection header received
0806 04:29:47.181076 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.181079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.181133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.181205 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.181812 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.182957 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.183834 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.184265 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.184369 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:47.186782 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:47.186796 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:47.186813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:47.186847 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:47.188368 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.188844 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.188852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.188855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.189046 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.189322 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.189326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.189328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.189640 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:47.189863 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:47.190086 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 04:29:47.190258 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 04:29:47.194629 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:47.193617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.28.94.1:45037)
0806 04:29:47.193748 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.193753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.193816 (+ 63us) server_negotiation.cc:407] Connection header received
0806 04:29:47.193956 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.193959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.194007 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.194080 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.194511 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53346: BlockingRecv error: recv got EOF from 127.0.0.1:53346 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 04:29:47.194751 29048 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.195871 29048 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.196765 29048 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:47.206063 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:47.197728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.28.94.1:45037)
0806 04:29:47.197860 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.197865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.197966 (+ 101us) server_negotiation.cc:407] Connection header received
0806 04:29:47.198111 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.198114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.198156 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.198223 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.198749 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.199291 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.200057 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.200289 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.200363 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:47.202786 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:47.202790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:47.202792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:47.202823 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:47.204318 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.204760 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.204764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.204766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.204862 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.205162 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.205166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.205168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.205371 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:47.205532 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:47.205767 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 04:29:47.205894 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:29:47.208948 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:47.208023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.28.94.1:45037)
0806 04:29:47.208167 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.208171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.208212 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:29:47.208317 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.208320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.208360 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.208421 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.208768 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53364: BlockingRecv error: recv got EOF from 127.0.0.1:53364 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:29:47.209275 29048 init.cc:244] Error while opening credential cache '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785990554567992-29048-0/krb5kdc/krb5cc)
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.210294 29048 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:47.211107 29048 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:29:47.220072 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:47.212019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.28.94.1:45037)
0806 04:29:47.212151 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:47.212156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:47.212264 (+ 108us) server_negotiation.cc:407] Connection header received
0806 04:29:47.212397 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:47.212400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:47.212444 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:47.212512 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:47.213092 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.213605 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.214407 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:47.214641 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:47.214731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:47.217095 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:47.217099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:47.217101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:47.217134 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:47.218382 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.218811 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.218814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.218815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.218899 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:47.219174 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:47.219177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:47.219179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:47.219375 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:47.219522 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:47.219786 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 04:29:47.219905 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:47.221273 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4243
I20260806 04:29:47.238073 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4436
I20260806 04:29:47.262312 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4619
I20260806 04:29:47.274210 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4111
2026-08-06T04:29:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2403 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4806](info): set up 2 sockets
Aug 06 04:29:47 dist-test-slave-ncp9 krb5kdc[4806](info): commencing operation
krb5kdc: starting...
W20260806 04:29:48.395207 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:48 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab.
Entry for principal oryx/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab.
I20260806 04:29:48.528662 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36675
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab
--principal=oryx/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:36675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:48.651407 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:48.651734 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:48.651778 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:48.656026 4822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:48.656111 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:48.656131 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:48.656147 4822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:48.656160 4822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:48.661998 4822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:36675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab
--principal=oryx/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:48.663239 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:48.664323 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:48.673568 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:48.673547 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:48.673550 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:48.673777 4822 server_base.cc:1034] running on GCE node
I20260806 04:29:48.674660 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:48.675946 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:48.677134 4822 hybrid_clock.cc:648] HybridClock initialized: now 1785990588677101 us; error 48 us; skew 500 ppm
Aug 06 04:29:48 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:48.680140 4822 init.cc:377] Logged in from keytab as oryx/127.28.94.62@KRBTEST.COM (short username oryx)
I20260806 04:29:48.681252 4822 webserver.cc:460] Webserver started at http://127.28.94.62:39993/ using document root <none> and password file <none>
I20260806 04:29:48.681895 4822 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:48.681972 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:48.682227 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:48.684504 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5c2e2b9d21dc48498dc684f2b1fa7010"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "ff113802641550dc6dcf993d7cca53a1"
server_key_iv: "4c5f61093cad84b30aef8419243f9e46"
server_key_version: "encryptionkey@0"
I20260806 04:29:48.685160 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5c2e2b9d21dc48498dc684f2b1fa7010"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "ff113802641550dc6dcf993d7cca53a1"
server_key_iv: "4c5f61093cad84b30aef8419243f9e46"
server_key_version: "encryptionkey@0"
I20260806 04:29:48.689575 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:29:48.692354 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:48.693756 4822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:48.693899 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5c2e2b9d21dc48498dc684f2b1fa7010"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "ff113802641550dc6dcf993d7cca53a1"
server_key_iv: "4c5f61093cad84b30aef8419243f9e46"
server_key_version: "encryptionkey@0"
I20260806 04:29:48.694021 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:48.729239 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:48.732807 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:48.733050 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:48.740615 4822 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:36675
I20260806 04:29:48.740916 4907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:36675 every 8 connection(s)
I20260806 04:29:48.741804 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:48.745005 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:48.747642 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4822
I20260806 04:29:48.747815 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:48.748167 29048 external_mini_cluster.cc:1428] Setting key d53b12284e3f7af647e5b31756e0798b
I20260806 04:29:48.752324 4909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Bootstrap starting.
I20260806 04:29:48.755331 4909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:48.756330 4909 log.cc:824] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:29:48 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:48.758929 4909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: No bootstrap required, opened a new log
I20260806 04:29:48.762799 4909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } }
I20260806 04:29:48.763131 4909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:48.763181 4909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2e2b9d21dc48498dc684f2b1fa7010, State: Initialized, Role: FOLLOWER
I20260806 04:29:48.763674 4909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } }
I20260806 04:29:48.763809 4909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:48.763872 4909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:48.763972 4909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:48.764122 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:48.749815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.28.94.62:36675)
0806 04:29:48.750296 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:48.750314 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:48.750354 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:29:48.751081 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:48.751101 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:48.751480 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:48.751804 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:48.752780 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:48.753926 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:48.755098 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:48.755672 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:48.755803 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:48.758827 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:48.758835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:48.758858 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:48.758893 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:48.760680 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:48.761227 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:48.761233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:48.761238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:48.761367 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:48.761671 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:48.761674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:48.761675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:48.762132 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:48.762359 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:48.762690 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 04:29:48.763090 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":148,"threads_started":1}
I20260806 04:29:48.765049 4909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } }
I20260806 04:29:48.765435 4909 leader_election.cc:302] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c2e2b9d21dc48498dc684f2b1fa7010; no voters:
I20260806 04:29:48.765810 4909 leader_election.cc:288] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:48.765918 4918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:48.766088 4918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 LEADER]: Becoming Leader. State: Replica: 5c2e2b9d21dc48498dc684f2b1fa7010, State: Running, Role: LEADER
I20260806 04:29:48.766430 4918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } }
I20260806 04:29:48.767267 4909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:48.767751 4918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2e2b9d21dc48498dc684f2b1fa7010. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } }
I20260806 04:29:48.767881 4918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:48.768343 4920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } }
I20260806 04:29:48.768455 4923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:48.768534 4920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:48.771751 4923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:48.778359 4923 catalog_manager.cc:1347] Generated new cluster ID: 5a202b740b8b44aebd774d9c936bbf58
I20260806 04:29:48.778460 4923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:48.793146 4923 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:48.794582 4923 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:48.803851 4923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Generated new TSK 0
I20260806 04:29:48.804854 4923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
I20260806 04:29:48.840368 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:48.965932 4946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:48.966269 4946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:48.966336 4946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:48.970860 4946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:48.970950 4946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:48.971038 4946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:48.976804 4946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:48.978091 4946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:48.979202 4946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:48.990011 4954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:48.991636 4952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:48.991596 4946 server_base.cc:1034] running on GCE node
W20260806 04:29:48.991715 4951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:48.992231 4946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:48.993000 4946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:48.994199 4946 hybrid_clock.cc:648] HybridClock initialized: now 1785990588994175 us; error 37 us; skew 500 ppm
Aug 06 04:29:48 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:48.997304 4946 init.cc:377] Logged in from keytab as oryx/127.28.94.1@KRBTEST.COM (short username oryx)
I20260806 04:29:48.998484 4946 webserver.cc:460] Webserver started at http://127.28.94.1:41905/ using document root <none> and password file <none>
I20260806 04:29:48.999179 4946 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:48.999255 4946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:48.999493 4946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:49.001850 4946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "581de6cb02f54a2d9dcc8a6fef115468"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "a4e662a8ef4372c0e3cd6675e0f01053"
server_key_iv: "8da60e0991fc850eb114c122ef79869a"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.002501 4946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "581de6cb02f54a2d9dcc8a6fef115468"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "a4e662a8ef4372c0e3cd6675e0f01053"
server_key_iv: "8da60e0991fc850eb114c122ef79869a"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.006835 4946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:29:49.010010 4963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.011617 4946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:49.011777 4946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "581de6cb02f54a2d9dcc8a6fef115468"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "a4e662a8ef4372c0e3cd6675e0f01053"
server_key_iv: "8da60e0991fc850eb114c122ef79869a"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.011912 4946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.026494 4946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:49.029995 4946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:49.030213 4946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:49.030893 4946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:49.031842 4946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:49.031909 4946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.031973 4946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:49.031999 4946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.041275 4946 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:46015
I20260806 04:29:49.041301 5119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:46015 every 8 connection(s)
I20260806 04:29:49.042328 4946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:49.049361 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 4946
I20260806 04:29:49.049494 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:49.049800 29048 external_mini_cluster.cc:1428] Setting key 8ecc4882c56958eac9e74c5fcada3a79
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:49.057678 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.044287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:47757 (local address 127.28.94.62:36675)
0806 04:29:49.044473 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.044478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.045430 (+ 952us) server_negotiation.cc:407] Connection header received
0806 04:29:49.046889 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.046892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.046951 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.047030 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.048256 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.048802 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.050074 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.050383 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.050497 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.053486 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.053491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.053494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.053531 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.055044 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.055753 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.055756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.055758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.055858 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.056279 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.056281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.056282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.056488 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.056568 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.057399 (+ 831us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.057535 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:29:49.058983 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.044740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:47757)
0806 04:29:49.045265 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.045313 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.046652 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.047185 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.047192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.047593 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.048123 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.048138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.048936 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.048939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.049934 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.049946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.050552 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.050557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.050766 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.051472 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.051493 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.053267 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.055187 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.055194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.055216 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.055597 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.055959 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.055962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.055964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.056187 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.056606 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.056613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.057066 (+ 453us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.057319 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.057743 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":159,"threads_started":1}
I20260806 04:29:49.060566 5120 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:49.060846 5120 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:29:49.061493 5120 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:49.063401 4863 ts_manager.cc:194] Registered new tserver with Master: 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.2@KRBTEST.COM" created.
I20260806 04:29:49.064733 4863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:47757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
I20260806 04:29:49.080595 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:49.205606 5129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:49.205924 5129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:49.205977 5129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:49.210247 5129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:49.210330 5129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:49.210393 5129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:49.216300 5129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:49.217851 5129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:49.218982 5129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:49.229710 5138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.229766 5136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.229718 5135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:49.230161 5129 server_base.cc:1034] running on GCE node
I20260806 04:29:49.230641 5129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:49.231366 5129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:49.232573 5129 hybrid_clock.cc:648] HybridClock initialized: now 1785990589232540 us; error 49 us; skew 500 ppm
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:49.235738 5129 init.cc:377] Logged in from keytab as oryx/127.28.94.2@KRBTEST.COM (short username oryx)
I20260806 04:29:49.236805 5129 webserver.cc:460] Webserver started at http://127.28.94.2:34821/ using document root <none> and password file <none>
I20260806 04:29:49.237469 5129 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:49.237560 5129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:49.237807 5129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:49.240068 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "c4ef7401105f413db2ec467b71b49acc"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "fc0809449ea7c71437b960fc8733988c"
server_key_iv: "73bdade389abcaa22adf6a0f229bef60"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.240675 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "c4ef7401105f413db2ec467b71b49acc"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "fc0809449ea7c71437b960fc8733988c"
server_key_iv: "73bdade389abcaa22adf6a0f229bef60"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.245191 5129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:29:49.248193 5149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.249697 5129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:49.249857 5129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "c4ef7401105f413db2ec467b71b49acc"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "fc0809449ea7c71437b960fc8733988c"
server_key_iv: "73bdade389abcaa22adf6a0f229bef60"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.249984 5129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.271136 5129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:49.274755 5129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:49.274957 5129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:49.275635 5129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:49.276575 5129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:49.276628 5129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.276677 5129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:49.276706 5129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.285634 5129 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42151
I20260806 04:29:49.285676 5301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42151 every 8 connection(s)
I20260806 04:29:49.286696 5129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:49.288966 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 5129
I20260806 04:29:49.289105 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:49.289381 29048 external_mini_cluster.cc:1428] Setting key d622236eb48ded3e1d934ad6ad19b2a6
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:29:49.303180 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.288825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:59945 (local address 127.28.94.62:36675)
0806 04:29:49.289018 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.289023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.290089 (+ 1066us) server_negotiation.cc:407] Connection header received
0806 04:29:49.291714 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.291718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.291779 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.291876 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.293302 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.293839 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.295286 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.295614 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.295727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.298814 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.298818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.298820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.298852 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.300469 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.301269 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.301273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.301274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.301367 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.301709 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.301713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.301715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.301977 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.302069 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.302835 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.303003 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.3@KRBTEST.COM" created.
I20260806 04:29:49.304060 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.289335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:59945)
0806 04:29:49.289915 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.289978 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.291451 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.292046 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.292054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.292468 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.293111 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.293138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.293983 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.293986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.295140 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.295152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.295819 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.295823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.295990 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.296643 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.296662 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.298583 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.300633 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.300642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.300666 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.301140 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.301449 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.301452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.301453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.301614 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.302124 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.302131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.302449 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.302636 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.302965 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":166,"threads_started":1}
I20260806 04:29:49.305516 5303 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:49.305847 5303 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:49.306442 5303 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:49.307695 4863 ts_manager.cc:194] Registered new tserver with Master: c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:49.308319 4863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:59945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
I20260806 04:29:49.324970 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:49.449961 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:49.450424 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:49.450510 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:49.455284 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:49.455374 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:49.455461 5309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:49.461442 5309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:49.462924 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:49.464128 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:49.475236 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:49.475298 5309 server_base.cc:1034] running on GCE node
W20260806 04:29:49.475240 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.475236 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:49.476055 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:49.476804 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:49.477973 5309 hybrid_clock.cc:648] HybridClock initialized: now 1785990589477947 us; error 49 us; skew 500 ppm
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:49.481098 5309 init.cc:377] Logged in from keytab as oryx/127.28.94.3@KRBTEST.COM (short username oryx)
I20260806 04:29:49.482111 5309 webserver.cc:460] Webserver started at http://127.28.94.3:42111/ using document root <none> and password file <none>
I20260806 04:29:49.482769 5309 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:49.482834 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:49.483048 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:49.485401 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "7988062833dc429f91442184b0969fc2"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "e818444d40e47a0b549f7274344fafbc"
server_key_iv: "b3619b52bcc62227f6779ab840107b56"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.486017 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "7988062833dc429f91442184b0969fc2"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "e818444d40e47a0b549f7274344fafbc"
server_key_iv: "b3619b52bcc62227f6779ab840107b56"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.490442 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:49.493430 5330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.494926 5309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:49.495081 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "7988062833dc429f91442184b0969fc2"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "e818444d40e47a0b549f7274344fafbc"
server_key_iv: "b3619b52bcc62227f6779ab840107b56"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.495214 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.510442 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:49.514031 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:49.514253 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:49.514935 5309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:49.515941 5309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:49.515995 5309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.516047 5309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:49.516076 5309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.525624 5309 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:33719
I20260806 04:29:49.525664 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:33719 every 8 connection(s)
I20260806 04:29:49.526657 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:49.533586 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 5309
I20260806 04:29:49.533732 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:49.534052 29048 external_mini_cluster.cc:1428] Setting key c2326e676ace50217eb5585e1e658596
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:49.541452 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.528714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:35651 (local address 127.28.94.62:36675)
0806 04:29:49.528879 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.528883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.529904 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 04:29:49.531393 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.531396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.531466 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.531541 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.532936 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.533515 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.534793 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.535022 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.535125 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.537774 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.537777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.537779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.537810 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.539160 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.539735 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.539738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.539740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.539836 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.540176 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.540180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.540182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.540427 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.540544 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.541170 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.541311 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:29:49.542443 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.529196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:35651)
0806 04:29:49.529727 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.529777 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.531141 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.531760 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.531768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.532178 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.532728 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.532745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.533658 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.533661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.534663 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.534675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.535189 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.535193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.535379 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.535976 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.535994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.537534 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.539285 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.539292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.539309 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.539614 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.539910 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.539912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.539914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.540065 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.540572 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.540579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.540885 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.541130 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.541461 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":147,"threads_started":1}
I20260806 04:29:49.543766 5496 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:49.544021 5496 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:49.544564 5496 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:49.545737 4863 ts_manager.cc:194] Registered new tserver with Master: 7988062833dc429f91442184b0969fc2 (127.28.94.3:33719)
I20260806 04:29:49.546325 4863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:35651
I20260806 04:29:49.548233 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.61.keytab.
Entry for principal oryx/127.28.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.61.keytab.
I20260806 04:29:49.580345 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:35759
--webserver_interface=127.28.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.61.keytab
--principal=oryx/127.28.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:36675,127.28.94.61:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:49.706149 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:49.706488 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:49.706532 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:49.710868 5504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:49.710958 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:49.710978 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:49.710994 5504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:49.711007 5504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:49.716794 5504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:36675,127.28.94.61:35759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.61.keytab
--principal=oryx/127.28.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
W20260806 04:29:49.717109 5504 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.94.62:36675,127.28.94.61:35759'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 04:29:49.718170 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:49.719276 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:49.728762 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.728762 5511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.728768 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:49.729198 5504 server_base.cc:1034] running on GCE node
I20260806 04:29:49.729873 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:49.731165 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:49.732354 5504 hybrid_clock.cc:648] HybridClock initialized: now 1785990589732322 us; error 61 us; skew 500 ppm
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:49.735598 5504 init.cc:377] Logged in from keytab as oryx/127.28.94.61@KRBTEST.COM (short username oryx)
I20260806 04:29:49.736713 5504 webserver.cc:460] Webserver started at http://127.28.94.61:34847/ using document root <none> and password file <none>
I20260806 04:29:49.737370 5504 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:49.737448 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:49.737692 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:49.739952 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/instance:
uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "709090d83165b593d72c4dece054f3e0"
server_key_iv: "7fad9c9b1868e88ca4fc4b57b6871182"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.740603 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal/instance:
uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "709090d83165b593d72c4dece054f3e0"
server_key_iv: "7fad9c9b1868e88ca4fc4b57b6871182"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.745024 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:29:49.747967 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.749536 5504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:49.749728 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "709090d83165b593d72c4dece054f3e0"
server_key_iv: "7fad9c9b1868e88ca4fc4b57b6871182"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.749847 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.771288 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:49.774688 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:49.774894 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:49.782512 5504 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.61:35759
I20260806 04:29:49.782790 5598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.61:35759 every 8 connection(s)
I20260806 04:29:49.783665 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
I20260806 04:29:49.786818 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:49.789295 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 5504
I20260806 04:29:49.791474 5601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.61@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
I20260806 04:29:49.803310 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.793483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.28.94.62:36675)
0806 04:29:49.793635 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.793639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.794078 (+ 439us) server_negotiation.cc:407] Connection header received
0806 04:29:49.794199 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.794201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.794248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.794314 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.795087 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.795600 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.796523 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.796743 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.796833 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.799402 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.799407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.799409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.799445 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.800885 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.801448 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.801450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.801452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.801530 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.801885 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.801889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.801891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.802223 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.802338 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.802967 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.803128 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:29:49.804456 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.793371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56126)
0806 04:29:49.793937 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.793981 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.794105 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.794385 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.794392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.794761 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.794929 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.794940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.795734 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.795737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.796403 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.796412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.796924 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.796929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.797026 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.797489 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.797506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.799167 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.801014 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.801021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.801035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.801351 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.801608 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.801611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.801613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.801798 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.802330 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.802335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.802697 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.802906 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.803283 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":164,"threads_started":1}
I20260806 04:29:49.804456 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.791021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.28.94.61:35759)
0806 04:29:49.791503 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.791520 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.791571 (+ 51us) server_negotiation.cc:407] Connection header received
0806 04:29:49.792289 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.792310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.792698 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.793069 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.793927 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.794969 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.795839 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.796349 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.796502 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.799774 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.799782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.799799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.799827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.801620 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.802102 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.802107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.802108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.802180 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.802468 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.802471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.802473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.802805 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.803008 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.803213 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.803341 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":2432,"thread_start_us":133,"threads_started":1}
I20260806 04:29:49.806157 5601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.61@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
I20260806 04:29:49.814288 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.806729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43866)
0806 04:29:49.806857 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.806871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.806947 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.807261 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.807263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.807416 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.807497 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.807502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.808757 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.808759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.809337 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.809345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.809822 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.809824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.809897 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.810354 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.810366 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.811604 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.813066 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.813069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.813071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.813260 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.813459 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.813462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.813463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.813526 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.813845 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.813847 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.813929 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.814005 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.814130 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:29:49.814316 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.806780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.28.94.61:35759)
0806 04:29:49.806894 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.806898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.806913 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:49.807043 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.807047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.807092 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.807165 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.807594 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.808663 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.809442 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.809725 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.809812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.811749 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.811753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.811755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.811778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.812957 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.813331 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.813333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.813334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.813402 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.813592 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.813595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.813597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.813782 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.813853 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.814053 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.814189 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":49}
I20260806 04:29:49.817576 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Bootstrap starting.
I20260806 04:29:49.820116 5601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:49.820979 5601 log.cc:824] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:49.823194 5601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: No bootstrap required, opened a new log
I20260806 04:29:49.826475 5601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } }
I20260806 04:29:49.826735 5601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:49.826771 5601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ff06d82d2d4901a399cd8fd9d3c85e, State: Initialized, Role: FOLLOWER
I20260806 04:29:49.827288 5601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } }
I20260806 04:29:49.827880 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } } }
I20260806 04:29:49.828016 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:49.828687 5601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:29:49.833982 5626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:29:49.834106 5626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:29:49.836593 5504 master_runner.cc:428] Detected that this master 00ff06d82d2d4901a399cd8fd9d3c85e is joining an existing cluster
I20260806 04:29:49.836668 5504 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.94.61:35759
I20260806 04:29:49.847234 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.838934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.28.94.61:35759)
0806 04:29:49.839066 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.839072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.839601 (+ 529us) server_negotiation.cc:407] Connection header received
0806 04:29:49.839753 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.839756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.839800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.839872 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.840469 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.841201 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.842064 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.842346 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.842433 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.843775 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.843779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.843781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.843803 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.845489 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.845987 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.845990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.845992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.846077 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.846344 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.846347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.846349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.846583 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.846666 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.846912 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.847055 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":42}
I20260806 04:29:49.847445 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.838911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56128)
0806 04:29:49.839386 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.839402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.839534 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.839792 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.839795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.840015 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.840135 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.840141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.841202 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.841205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.842304 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.842315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.842891 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.842895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.842990 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.843393 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.843407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.843817 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.845678 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.845682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.845685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.846015 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.846310 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.846313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.846315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.846405 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.846855 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.846858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.846939 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.847052 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.847180 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":82,"thread_start_us":122,"threads_started":1}
I20260806 04:29:49.847707 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.839200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43872)
0806 04:29:49.839522 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.839535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.839659 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.839976 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.839979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.840227 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.840356 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.840363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.841303 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.841306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.841947 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.841956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.842580 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.842583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.842663 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.843194 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.843211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.843650 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.845623 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.845627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.845629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.845899 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.846151 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.846155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.846157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.846239 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.846663 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.846666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.846760 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.847419 (+ 659us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.847550 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":21,"thread_start_us":122,"threads_started":1}
I20260806 04:29:49.847955 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.839068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.28.94.62:36675)
0806 04:29:49.839176 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.839181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.839493 (+ 312us) server_negotiation.cc:407] Connection header received
0806 04:29:49.839652 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.839655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.839711 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.839783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.840254 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.841084 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.842435 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.842764 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.843162 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.843967 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.843971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.843973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.844004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.845569 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.846128 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.846132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.846134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.846224 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.846514 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.846518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.846520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.846758 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.846861 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.847116 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.847248 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.62@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
I20260806 04:29:49.862792 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.853032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.28.94.61:35759)
0806 04:29:49.853151 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.853155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.853259 (+ 104us) server_negotiation.cc:407] Connection header received
0806 04:29:49.853523 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.853526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.853608 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.853688 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.854394 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.855233 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.856109 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.856446 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.856553 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.859098 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.859102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.859104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.859128 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.860623 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.861245 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.861249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.861251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.861351 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.861667 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.861670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.861672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.861914 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.862007 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.862458 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.862610 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:29:49.862872 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.852551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43878)
0806 04:29:49.852991 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.853055 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.853333 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.853749 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.853755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.854164 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.854287 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.854294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.855339 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.855342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.855995 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.856005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.856554 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.856556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.856639 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.857238 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.857251 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.858942 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.860742 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.860756 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.860766 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.861140 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.861457 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.861459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.861461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.861571 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.862017 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.862025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.862208 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.862379 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.862638 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
I20260806 04:29:49.864527 4863 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.94.61:35759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
member_type: NON_VOTER
last_known_addr {
host: "127.28.94.61"
port: 35759
}
attrs {
promote: true
}
}
dest_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010"
cas_config_opid_index: -1
I20260806 04:29:49.865334 4863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } }
I20260806 04:29:49.867406 5643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } }
I20260806 04:29:49.868389 5563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:49.869468 5643 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.94.61:35759
I20260806 04:29:49.869879 5563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 04:29:49.869855 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } }
I20260806 04:29:49.870005 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:49.870668 5504 master.cc:537] Master@127.28.94.61:35759 shutting down...
I20260806 04:29:49.870695 5642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.28.94.60@KRBTEST.COM; defaulting to no policy
W20260806 04:29:49.873205 4846 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 04:29:49.874460 4846 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 -> Peer 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61:35759): Couldn't send request to peer 00ff06d82d2d4901a399cd8fd9d3c85e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:29:49.875912 5504 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:49.876251 5504 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:49.876375 5504 tablet_replica.cc:330] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.60@KRBTEST.COM" created.
I20260806 04:29:49.892565 5504 master.cc:552] Master@127.28.94.61:35759 shutdown complete.
I20260806 04:29:49.892652 5504 master_runner.cc:305] Clearing existing system tablet
I20260806 04:29:49.892978 5504 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:49.893046 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:49.894603 5504 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:49.895514 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.895797 5504 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.895954 5504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:49.895998 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "709090d83165b593d72c4dece054f3e0"
server_key_iv: "7fad9c9b1868e88ca4fc4b57b6871182"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.898558 5504 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:49.901042 5504 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:29:49.901186 5504 log.cc:1196] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 04:29:49.901429 5504 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Deleting consensus metadata
I20260806 04:29:49.902382 5504 master_runner.cc:315] Copying system tablet from 127.28.94.62:36675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.60.keytab.
Entry for principal oryx/127.28.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.60.keytab.
I20260806 04:29:49.903789 5504 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: tablet copy: Beginning tablet copy session from remote peer at address 127.28.94.62:36675
I20260806 04:29:49.904184 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:38615
--webserver_interface=127.28.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.60.keytab
--principal=oryx/127.28.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:29:49.913802 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.904683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.28.94.62:36675)
0806 04:29:49.904821 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:49.904825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:49.905011 (+ 186us) server_negotiation.cc:407] Connection header received
0806 04:29:49.905159 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:49.905162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:49.905220 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:49.905316 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:49.906232 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.907076 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.908145 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.908499 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.908618 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.910059 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:49.910064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:49.910067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:49.910101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:49.911804 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.912366 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.912370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.912372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.912470 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:49.912790 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:49.912794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:49.912795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:49.913072 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:49.913175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:49.913494 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 04:29:49.913640 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:49.914220 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:49.904524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56138)
0806 04:29:49.904876 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:49.904923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:49.905064 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:49.905623 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:49.905626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:49.905866 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:49.906027 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.906037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.907232 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.907237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.908022 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:49.908034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:49.908652 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:49.908656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:49.908752 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:49.909381 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:49.909397 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:49.909876 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:49.911946 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.911951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.911953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.912249 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.912566 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:49.912569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:49.912571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:49.912694 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:49.913176 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:49.913179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:49.913311 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 04:29:49.913435 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 04:29:49.913618 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260806 04:29:49.914999 4887 tablet_copy_service.cc:140] P 5c2e2b9d21dc48498dc684f2b1fa7010: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 00ff06d82d2d4901a399cd8fd9d3c85e ({username='oryx', principal='oryx/127.28.94.61@KRBTEST.COM'} at 127.0.0.1:56138)
I20260806 04:29:49.915233 4887 tablet_copy_service.cc:161] P 5c2e2b9d21dc48498dc684f2b1fa7010: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 00ff06d82d2d4901a399cd8fd9d3c85e at {username='oryx', principal='oryx/127.28.94.61@KRBTEST.COM'} at 127.0.0.1:56138: session id = 00ff06d82d2d4901a399cd8fd9d3c85e-00000000000000000000000000000000
I20260806 04:29:49.917448 4887 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:29:49.919721 5504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:29:49.922621 5504 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: tablet copy: Starting download of 0 data blocks...
I20260806 04:29:49.922904 5504 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: tablet copy: Starting download of 1 WAL segments...
I20260806 04:29:49.925472 5504 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:29:49.927837 4887 tablet_copy_service.cc:342] P 5c2e2b9d21dc48498dc684f2b1fa7010: Request end of tablet copy session 00ff06d82d2d4901a399cd8fd9d3c85e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.94.61@KRBTEST.COM'} at 127.0.0.1:56138
I20260806 04:29:49.928028 4887 tablet_copy_service.cc:433] P 5c2e2b9d21dc48498dc684f2b1fa7010: ending tablet copy session 00ff06d82d2d4901a399cd8fd9d3c85e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 00ff06d82d2d4901a399cd8fd9d3c85e
I20260806 04:29:49.937068 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:49.938905 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.939141 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:49.939321 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:49.939955 5504 server_base.cc:1034] running on GCE node
I20260806 04:29:49.940128 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:49.940466 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:49.941612 5504 hybrid_clock.cc:648] HybridClock initialized: now 1785990589941595 us; error 42 us; skew 500 ppm
Aug 06 04:29:49 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990589, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:49.943248 5504 init.cc:377] Logged in from keytab as oryx/127.28.94.61@KRBTEST.COM (short username oryx)
I20260806 04:29:49.943782 5504 webserver.cc:460] Webserver started at http://127.28.94.61:44659/ using document root <none> and password file <none>
I20260806 04:29:49.943993 5504 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:49.944038 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:49.945544 5504 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 04:29:49.946708 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:49.947010 5504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:49.947094 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "709090d83165b593d72c4dece054f3e0"
server_key_iv: "7fad9c9b1868e88ca4fc4b57b6871182"
server_key_version: "encryptionkey@0"
I20260806 04:29:49.947180 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:49.970732 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:49.973207 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:49.978701 5504 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.61:35759
I20260806 04:29:49.978740 5748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.61:35759 every 8 connection(s)
I20260806 04:29:49.979579 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
I20260806 04:29:49.981813 5749 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:29:49.982980 5749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Bootstrap starting.
I20260806 04:29:49.996752 5749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:29:49.997156 5749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: Bootstrap complete.
I20260806 04:29:49.997844 5749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } }
I20260806 04:29:49.997992 5749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00ff06d82d2d4901a399cd8fd9d3c85e, State: Initialized, Role: LEARNER
I20260806 04:29:49.998090 5749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } }
I20260806 04:29:49.998589 5749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:49.998561 5753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } }
I20260806 04:29:49.998946 5753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: LEARNER
I20260806 04:29:50.002593 5770 catalog_manager.cc:1259] Loaded cluster ID: 5a202b740b8b44aebd774d9c936bbf58
I20260806 04:29:50.002681 5770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: loading cluster ID for follower catalog manager: success
I20260806 04:29:50.004994 5770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: acquiring CA information for follower catalog manager: success
I20260806 04:29:50.006745 5770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 04:29:50.057258 5661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:50.057582 5661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:50.057626 5661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:50.061859 5661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:50.061945 5661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:50.061965 5661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:50.061982 5661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:50.061996 5661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:50.067981 5661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.60.keytab
--principal=oryx/127.28.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:50.068209 5120 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
I20260806 04:29:50.069423 5661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:50.070690 5661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:50.080205 5780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.080185 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.080251 5661 server_base.cc:1034] running on GCE node
W20260806 04:29:50.080185 5775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.081192 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:50.082430 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:50.083643 5661 hybrid_clock.cc:648] HybridClock initialized: now 1785990590083612 us; error 44 us; skew 500 ppm
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:50.086812 5661 init.cc:377] Logged in from keytab as oryx/127.28.94.60@KRBTEST.COM (short username oryx)
I20260806 04:29:50.087908 5661 webserver.cc:460] Webserver started at http://127.28.94.60:36541/ using document root <none> and password file <none>
I20260806 04:29:50.088496 5661 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.088560 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.088762 5661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:50.091109 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/instance:
uuid: "296192dc335149b2a6e28c61fcb2f71d"
format_stamp: "Formatted at 2026-08-06 04:29:50 on dist-test-slave-ncp9"
server_key: "fe6a3a8682aa8eebb36eaa987b575227"
server_key_iv: "5663764fbd5a4d58034c2675a616139c"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.091706 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal/instance:
uuid: "296192dc335149b2a6e28c61fcb2f71d"
format_stamp: "Formatted at 2026-08-06 04:29:50 on dist-test-slave-ncp9"
server_key: "fe6a3a8682aa8eebb36eaa987b575227"
server_key_iv: "5663764fbd5a4d58034c2675a616139c"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.097080 5661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 04:29:50.100733 5791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.104230 5661 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 04:29:50.104415 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "296192dc335149b2a6e28c61fcb2f71d"
format_stamp: "Formatted at 2026-08-06 04:29:50 on dist-test-slave-ncp9"
server_key: "fe6a3a8682aa8eebb36eaa987b575227"
server_key_iv: "5663764fbd5a4d58034c2675a616139c"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.104590 5661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.116936 5661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:50.120597 5661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:50.120820 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:50.131882 5661 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.60:38615
I20260806 04:29:50.132186 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.60:38615 every 8 connection(s)
I20260806 04:29:50.133131 5661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
I20260806 04:29:50.134506 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 5661
I20260806 04:29:50.136463 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:50.140852 5871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.60@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:50.150077 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.136207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.28.94.60:38615)
0806 04:29:50.136660 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.136686 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.136727 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:29:50.137531 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.137557 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.137915 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.138255 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.139121 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.140202 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.141131 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.141683 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.141839 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.144834 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.144839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.144849 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.144880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.146709 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.147188 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.147194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.147199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.147320 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.147687 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.147690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.147692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.148197 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.148465 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.148831 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.149174 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
I20260806 04:29:50.152700 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.143163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.28.94.62:36675)
0806 04:29:50.143300 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.143305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.143670 (+ 365us) server_negotiation.cc:407] Connection header received
0806 04:29:50.143813 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.143816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.143867 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.143936 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.144751 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.145421 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.146414 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.146645 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.146739 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.149289 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.149292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.149294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.149335 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.150891 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.151349 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.151353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.151355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.151460 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.151748 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.151752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.151753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.151990 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.152084 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.152405 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.152531 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:50.152788 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.142947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56146)
0806 04:29:50.143525 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.143569 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.143718 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.144012 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.144019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.144422 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.144598 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.144605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.145525 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.145529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.146285 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.146297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.146756 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.146760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.146891 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.147361 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.147373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.149138 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.151014 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.151023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.151027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.151258 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.151573 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.151577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.151578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.151662 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.152094 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.152099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.152209 (+ 110us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.152319 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.152510 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
I20260806 04:29:50.153909 5871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.60@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
I20260806 04:29:50.162744 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.154461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43892)
0806 04:29:50.154590 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.154603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.154680 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.155139 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.155142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.155307 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.155431 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.155437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.156347 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.156351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.156915 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.156926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.157381 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.157384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.157521 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.158006 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.158017 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.159281 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.161177 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.161181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.161182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.161392 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.161708 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.161711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.161712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.161795 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.162213 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.162216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.162328 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.162436 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.162572 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:29:50.162750 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.154606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.28.94.61:35759)
0806 04:29:50.154933 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.154937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.154948 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:29:50.154985 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.154988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.155027 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.155105 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.155548 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.156236 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.157023 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.157274 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.157369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.159440 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.159444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.159446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.159474 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.161042 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.161516 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.161519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.161521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.161622 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.161862 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.161864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.161865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.162066 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.162161 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.162484 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.162608 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":150,"threads_started":1}
I20260806 04:29:50.163803 5871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.60@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
I20260806 04:29:50.172456 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.164363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.0.0.1:43816)
0806 04:29:50.164506 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.164518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.164599 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.164834 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.164837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.165029 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.165109 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.165115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.166545 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.166549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.167095 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.167105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.167645 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.167648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.167733 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.168197 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.168209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.169596 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.171024 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.171028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.171030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.171222 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.171469 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.171472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.171474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.171549 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.171909 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.171912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.172004 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.172090 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.172252 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 04:29:50.172456 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.164501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.28.94.60:38615)
0806 04:29:50.164606 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.164610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.164622 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:29:50.164710 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.164713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.164749 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.164810 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.165213 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.166299 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.167243 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.167530 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.167617 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.169732 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.169736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.169738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.169767 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.170919 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.171309 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.171312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.171314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.171391 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.171625 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.171628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.171630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.171822 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.171909 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.172148 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.172311 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":46}
I20260806 04:29:50.176044 5871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Bootstrap starting.
I20260806 04:29:50.178881 5871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:50.179780 5871 log.cc:824] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:50.182051 5871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: No bootstrap required, opened a new log
I20260806 04:29:50.185689 5871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } }
I20260806 04:29:50.185947 5871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:50.186018 5871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296192dc335149b2a6e28c61fcb2f71d, State: Initialized, Role: FOLLOWER
I20260806 04:29:50.186487 5871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } }
I20260806 04:29:50.187048 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } } }
I20260806 04:29:50.187186 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:50.188133 5871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:29:50.194859 5893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:29:50.194979 5893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:29:50.198117 5661 master_runner.cc:428] Detected that this master 296192dc335149b2a6e28c61fcb2f71d is joining an existing cluster
I20260806 04:29:50.198194 5661 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.94.60:38615
I20260806 04:29:50.203935 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4946
I20260806 04:29:50.209757 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.202043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.28.94.60:38615)
0806 04:29:50.202150 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.202154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.202398 (+ 244us) server_negotiation.cc:407] Connection header received
0806 04:29:50.202524 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.202527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.202560 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.202604 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.203159 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.203874 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.204825 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.205075 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.205136 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.206800 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.206804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.206807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.206825 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.208148 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.208571 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.208574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.208575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.208634 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.208951 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.208953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.208955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.209158 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.209215 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.209480 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.209595 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":41}
I20260806 04:29:50.210168 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.201760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.0.0.1:43828)
0806 04:29:50.202329 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.202340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.202445 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.202748 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.202750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.202931 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.203047 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.203054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.203993 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.203996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.204726 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.204735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.205267 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.205269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.205331 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.206263 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.206277 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.206674 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.208284 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.208286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.208288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.208495 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.208704 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.208706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.208708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.208764 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.209254 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.209255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.209361 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.209900 (+ 539us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.210015 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260806 04:29:50.230494 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.201163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.28.94.62:36675)
0806 04:29:50.201302 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.201307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.201854 (+ 547us) server_negotiation.cc:407] Connection header received
0806 04:29:50.202058 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.202063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.202120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.202178 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.203046 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.203903 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.224073 (+ 20170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.224411 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.224499 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.226286 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.226290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.226293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.226334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.228156 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.228697 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.228701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.228703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.228797 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.229172 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.229176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.229178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.229412 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.229483 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.230148 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.230310 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:29:50.231542 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.201396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56162)
0806 04:29:50.201754 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.201767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.201935 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.202354 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.202357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.202646 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.202896 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.202904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.204026 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.204029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.223817 (+ 19788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.223850 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.224736 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.224740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.224845 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.225535 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.225551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.226091 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.228293 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.228297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.228299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.228548 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.228938 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.228941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.228943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.229067 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.229792 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.229796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.229927 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.230896 (+ 969us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.231062 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":46,"thread_start_us":87,"threads_started":1}
I20260806 04:29:50.234037 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.201216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.28.94.61:35759)
0806 04:29:50.201331 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.201336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.201783 (+ 447us) server_negotiation.cc:407] Connection header received
0806 04:29:50.201990 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.201994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.202049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.202101 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.225579 (+ 23478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.226430 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.228572 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.228968 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.229055 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.230426 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.230431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.230434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.230465 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.232117 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.232624 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.232628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.232630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.232723 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.233060 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.233064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.233066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.233304 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.233376 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.233726 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.233875 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:29:50.234602 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.201181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43900)
0806 04:29:50.201689 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.201704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.201871 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.202298 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.202301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.202585 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.225313 (+ 22728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.225330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.227455 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.227459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.228407 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.228419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.229267 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.229270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.229376 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.229815 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.229829 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.230249 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.232262 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.232265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.232267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.232513 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.232838 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.232841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.232843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.232956 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.233448 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.233451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.233545 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.234251 (+ 706us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.234414 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":57,"thread_start_us":120,"threads_started":1}
I20260806 04:29:50.235242 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:46015
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:36675
--webserver_port=41905
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990588, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.62@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
I20260806 04:29:50.269228 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.236652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.0.0.1:43836)
0806 04:29:50.236841 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.236855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.237026 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.237402 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.237406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.237632 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.237761 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.237769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.238754 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.238758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.239659 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.239672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.240288 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.240291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.240402 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.241062 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.241079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.242778 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.267046 (+ 24268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.267052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.267054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.267404 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.267855 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.267859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.267861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.267973 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.268465 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.268469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.268626 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.268781 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.269024 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 04:29:50.269264 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.236724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.28.94.60:38615)
0806 04:29:50.236841 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.236846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.236937 (+ 91us) server_negotiation.cc:407] Connection header received
0806 04:29:50.237195 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.237198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.237257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.237329 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.237891 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.238635 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.239803 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.240151 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.240276 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.242960 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.242965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.242968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.243002 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.266835 (+ 23833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.267537 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.267541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.267543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.267738 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.268086 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.268089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.268091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.268360 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.268465 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.268860 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.269048 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:50.270918 4863 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.94.60:38615: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d"
member_type: NON_VOTER
last_known_addr {
host: "127.28.94.60"
port: 38615
}
attrs {
promote: true
}
}
dest_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010"
cas_config_opid_index: 5
I20260806 04:29:50.271646 4863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } }
I20260806 04:29:50.273100 5642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.273716 5829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:50.274783 5642 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.94.60:38615
I20260806 04:29:50.274982 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.275079 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:50.275244 5829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:29:50.275980 5661 master.cc:537] Master@127.28.94.60:38615 shutting down...
I20260806 04:29:50.279386 5643 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:50.281651 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.276353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.28.94.61:35759)
0806 04:29:50.276493 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.276497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.276676 (+ 179us) server_negotiation.cc:407] Connection header received
0806 04:29:50.276947 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.276951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.277027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.277100 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:50.277575 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.278382 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.279996 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.280665 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.280758 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.280967 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.281408 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.281504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:29:50.282130 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.275436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43914)
0806 04:29:50.276602 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.276618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.276773 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.277302 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.277306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.277331 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:50.277439 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.277446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.278545 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.278548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.279827 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.279841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.281082 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.281086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.281206 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.281279 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.281897 (+ 618us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.282002 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081}
I20260806 04:29:50.282855 5713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 LEARNER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:29:50.283420 5908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:29:50.286561 5713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.289021 5661 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 04:29:50.289338 4843 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 -> Peer 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60:38615): Couldn't send request to peer 296192dc335149b2a6e28c61fcb2f71d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:29:50.289386 5661 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:50.289553 5661 tablet_replica.cc:330] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: stopping tablet replica
I20260806 04:29:50.290802 5753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2e2b9d21dc48498dc684f2b1fa7010. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.290968 5753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: LEARNER
I20260806 04:29:50.291081 5753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: NON_VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: true } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.291170 5753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: LEARNER
I20260806 04:29:50.292956 5643 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: attempting to promote NON_VOTER 00ff06d82d2d4901a399cd8fd9d3c85e to VOTER
I20260806 04:29:50.293509 5643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } }
W20260806 04:29:50.296021 4843 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 -> Peer 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60:38615): Couldn't send request to peer 296192dc335149b2a6e28c61fcb2f71d. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:29:50.296161 5713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 LEARNER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 04:29:50.296694 5642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 04:29:50.297996 5661 master.cc:552] Master@127.28.94.60:38615 shutdown complete.
I20260806 04:29:50.298055 5661 master_runner.cc:305] Clearing existing system tablet
I20260806 04:29:50.298377 5661 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.298446 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.299229 5642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.299872 5661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:50.301195 5643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.301321 5643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:50.301544 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.301718 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.301762 5661 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.301846 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:50.301887 5661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:29:50.301923 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "296192dc335149b2a6e28c61fcb2f71d"
format_stamp: "Formatted at 2026-08-06 04:29:50 on dist-test-slave-ncp9"
server_key: "fe6a3a8682aa8eebb36eaa987b575227"
server_key_iv: "5663764fbd5a4d58034c2675a616139c"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.302219 5713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.304267 5661 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:50.304441 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.304580 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:50.306986 5661 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:29:50.307102 5661 log.cc:1196] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 04:29:50.307394 5661 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Deleting consensus metadata
I20260806 04:29:50.307565 5661 master_runner.cc:315] Copying system tablet from 127.28.94.62:36675
I20260806 04:29:50.308933 5661 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.94.62:36675
I20260806 04:29:50.310709 5303 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
I20260806 04:29:50.318389 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.309697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.28.94.62:36675)
0806 04:29:50.309861 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.309867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.309978 (+ 111us) server_negotiation.cc:407] Connection header received
0806 04:29:50.310103 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.310107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.310164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.310262 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.310799 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.311575 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.312926 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.313270 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.313385 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.314768 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.314773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.314776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.314806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.316494 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.317024 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.317028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.317030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.317133 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.317422 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.317425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.317427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.317686 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.317788 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.318080 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.318220 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:29:50.318400 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.309546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56176)
0806 04:29:50.309873 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.309888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.310014 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.310287 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.310290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.310524 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.310648 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.310655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.311824 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.311827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.312751 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.312764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.313402 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.313406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.313505 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.314094 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.314109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.314587 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.316622 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.316626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.316628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.316921 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.317226 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.317229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.317231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.317331 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.317786 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.317790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.317908 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.318020 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.318208 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260806 04:29:50.319336 4887 tablet_copy_service.cc:140] P 5c2e2b9d21dc48498dc684f2b1fa7010: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 296192dc335149b2a6e28c61fcb2f71d ({username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:56176)
I20260806 04:29:50.319456 4887 tablet_copy_service.cc:161] P 5c2e2b9d21dc48498dc684f2b1fa7010: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 296192dc335149b2a6e28c61fcb2f71d at {username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:56176: session id = 296192dc335149b2a6e28c61fcb2f71d-00000000000000000000000000000000
I20260806 04:29:50.320628 4887 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:29:50.322247 5661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:29:50.325013 5661 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: tablet copy: Starting download of 0 data blocks...
I20260806 04:29:50.325352 5661 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: tablet copy: Starting download of 1 WAL segments...
I20260806 04:29:50.327348 5661 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:29:50.329808 4887 tablet_copy_service.cc:342] P 5c2e2b9d21dc48498dc684f2b1fa7010: Request end of tablet copy session 296192dc335149b2a6e28c61fcb2f71d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:56176
I20260806 04:29:50.329927 4887 tablet_copy_service.cc:433] P 5c2e2b9d21dc48498dc684f2b1fa7010: ending tablet copy session 296192dc335149b2a6e28c61fcb2f71d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 296192dc335149b2a6e28c61fcb2f71d
I20260806 04:29:50.338850 5661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:50.340658 5931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.340742 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.341315 5661 server_base.cc:1034] running on GCE node
W20260806 04:29:50.341495 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.341779 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:50.342120 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:50.343264 5661 hybrid_clock.cc:648] HybridClock initialized: now 1785990590343245 us; error 38 us; skew 500 ppm
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:50.345089 5661 init.cc:377] Logged in from keytab as oryx/127.28.94.60@KRBTEST.COM (short username oryx)
I20260806 04:29:50.345639 5661 webserver.cc:460] Webserver started at http://127.28.94.60:33645/ using document root <none> and password file <none>
I20260806 04:29:50.345850 5661 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.345897 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.347640 5661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 04:29:50.348986 5940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.349279 5661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:50.349411 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "296192dc335149b2a6e28c61fcb2f71d"
format_stamp: "Formatted at 2026-08-06 04:29:50 on dist-test-slave-ncp9"
server_key: "fe6a3a8682aa8eebb36eaa987b575227"
server_key_iv: "5663764fbd5a4d58034c2675a616139c"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.349581 5661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.368039 5661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:50.370505 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:50.376163 5661 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.60:38615
I20260806 04:29:50.377084 5661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
I20260806 04:29:50.377877 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.60:38615 every 8 connection(s)
I20260806 04:29:50.379359 6018 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:29:50.380488 6018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Bootstrap starting.
I20260806 04:29:50.390659 6018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:29:50.390990 6018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Bootstrap complete.
I20260806 04:29:50.391646 6018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } }
I20260806 04:29:50.391800 6018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 296192dc335149b2a6e28c61fcb2f71d, State: Initialized, Role: LEARNER
I20260806 04:29:50.391906 6018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } }
I20260806 04:29:50.392277 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.392369 6018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:50.392441 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: LEARNER
I20260806 04:29:50.395324 6048 catalog_manager.cc:1259] Loaded cluster ID: 5a202b740b8b44aebd774d9c936bbf58
I20260806 04:29:50.395500 6048 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: loading cluster ID for follower catalog manager: success
I20260806 04:29:50.397284 6048 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: acquiring CA information for follower catalog manager: success
W20260806 04:29:50.397585 5907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:50.397888 5907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:50.397946 5907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:29:50.398756 6048 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 04:29:50.402712 5907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:50.402796 5907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:50.402871 5907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:50.408793 5907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:46015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=41905
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:50.410120 5907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:50.411293 5907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:29:50.422355 5907 server_base.cc:1034] running on GCE node
W20260806 04:29:50.422518 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.422549 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.422595 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.423210 5907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:50.423930 5907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:50.425099 5907 hybrid_clock.cc:648] HybridClock initialized: now 1785990590425078 us; error 42 us; skew 500 ppm
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:50.428059 5907 init.cc:377] Logged in from keytab as oryx/127.28.94.1@KRBTEST.COM (short username oryx)
I20260806 04:29:50.429154 5907 webserver.cc:460] Webserver started at http://127.28.94.1:41905/ using document root <none> and password file <none>
I20260806 04:29:50.429772 5907 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.429862 5907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.435487 5907 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:29:50.437970 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.439440 5907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:29:50.439577 5907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "581de6cb02f54a2d9dcc8a6fef115468"
format_stamp: "Formatted at 2026-08-06 04:29:48 on dist-test-slave-ncp9"
server_key: "a4e662a8ef4372c0e3cd6675e0f01053"
server_key_iv: "8da60e0991fc850eb114c122ef79869a"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.440213 5907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.454891 5907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:50.458221 5907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:50.458410 5907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:50.459054 5907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:50.460136 5907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:50.460199 5907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.460244 5907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:50.460265 5907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.473711 5907 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:46015
I20260806 04:29:50.473726 6233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:46015 every 8 connection(s)
I20260806 04:29:50.474917 5907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:50.479383 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 5907
I20260806 04:29:50.479600 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 5129
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
I20260806 04:29:50.508335 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.478992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:58501 (local address 127.28.94.60:38615)
0806 04:29:50.479362 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.479366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.479381 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:50.480272 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.480276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.480329 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.480406 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.482665 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.484132 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.488956 (+ 4824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.489440 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.489586 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.499638 (+ 10052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.499644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.499648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.499690 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.504215 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.504929 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.504933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.504935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.505046 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.505665 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.505670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.505672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.505991 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.506119 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.507956 (+ 1837us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.508130 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":180,"threads_started":1}
I20260806 04:29:50.509115 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.478887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.28.94.1:58501)
0806 04:29:50.479263 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.479276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.479626 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.480523 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.480526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.482317 (+ 1791us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.482488 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.482499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.484427 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.484432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.486624 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.486647 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.492756 (+ 6109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.492774 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.493123 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.495837 (+ 2714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.495861 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.498250 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.504401 (+ 6151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.504405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.504408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.504745 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.505201 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.505205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.505208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.505523 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.506741 (+ 1218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.506751 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.507195 (+ 444us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.507428 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.507854 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":77,"threads_started":1}
I20260806 04:29:50.509804 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.477068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:58213 (local address 127.28.94.61:35759)
0806 04:29:50.477187 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.477191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.478677 (+ 1486us) server_negotiation.cc:407] Connection header received
0806 04:29:50.479899 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.479903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.479962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.480013 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.486882 (+ 6869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.487887 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.490739 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.491204 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.491317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.499118 (+ 7801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.499127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.499131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.499179 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.501824 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.506237 (+ 4413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.506242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.506244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.506396 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.508695 (+ 2299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.508700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.508701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.509019 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.509121 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.509477 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.509642 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:29:50.510301 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.477597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.28.94.1:58213)
0806 04:29:50.478499 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.478546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.479644 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.480447 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.480454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.481200 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.481967 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.481996 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.488087 (+ 6091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.488091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.490531 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.490549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.491654 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.491662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.492034 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.493711 (+ 1677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.493747 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.498853 (+ 5106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.502549 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.502562 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.502592 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.503197 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.506528 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.506531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.506532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.506627 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.509187 (+ 2560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.509190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.509292 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.509954 (+ 662us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.510109 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":187,"threads_started":1}
I20260806 04:29:50.511713 6236 heartbeater.cc:347] Connected to a master server at 127.28.94.61:35759
I20260806 04:29:50.511848 6234 heartbeater.cc:347] Connected to a master server at 127.28.94.60:38615
I20260806 04:29:50.512010 6236 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.512010 6234 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.512753 6236 heartbeater.cc:511] Master 127.28.94.61:35759 requested a full tablet report, sending...
I20260806 04:29:50.512765 6234 heartbeater.cc:511] Master 127.28.94.60:38615 requested a full tablet report, sending...
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:50.514674 5966 ts_manager.cc:194] Registered new tserver with Master: 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015)
I20260806 04:29:50.514676 5691 ts_manager.cc:194] Registered new tserver with Master: 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015)
I20260806 04:29:50.516738 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:42151
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:36675
--webserver_port=34821
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:29:50.517467 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.476979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:39519 (local address 127.28.94.62:36675)
0806 04:29:50.477155 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.477160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.478938 (+ 1778us) server_negotiation.cc:407] Connection header received
0806 04:29:50.479887 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.479891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.479948 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.479995 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.508640 (+ 28645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.509451 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.510573 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.510889 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.510974 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.513883 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.513888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.513891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.513927 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.515695 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.516160 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.516164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.516166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.516253 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.516537 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.516541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.516543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.516763 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.516860 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.517227 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.517340 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:29:50.517522 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.477597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:39519)
0806 04:29:50.478845 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.478858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.479626 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.480443 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.480453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.508306 (+ 27853us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.508466 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.508475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.509597 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.509601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.510413 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.510425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.511191 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.511195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.511304 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.512123 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.512138 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.513707 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.515824 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.515828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.515830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.516032 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.516353 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.516356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.516358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.516441 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.516913 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.516917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.517051 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.517155 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.517324 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":25,"spinlock_wait_cycles":48512,"thread_start_us":161,"threads_started":1}
I20260806 04:29:50.518358 6235 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:50.518489 6235 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.518725 6235 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:50.519749 4863 ts_manager.cc:194] Re-registered known tserver with Master: 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015)
I20260806 04:29:50.520471 4863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:39519
I20260806 04:29:50.548810 5496 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
W20260806 04:29:50.640810 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:50.641165 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:50.641227 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:50.645591 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:50.645676 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:50.645761 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:50.651677 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:42151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=34821
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:50.652976 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:50.654127 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:50.664752 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.664866 6247 server_base.cc:1034] running on GCE node
W20260806 04:29:50.664760 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.664753 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.665550 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:50.666306 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:50.667553 6247 hybrid_clock.cc:648] HybridClock initialized: now 1785990590667518 us; error 43 us; skew 500 ppm
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:50.670912 6247 init.cc:377] Logged in from keytab as oryx/127.28.94.2@KRBTEST.COM (short username oryx)
I20260806 04:29:50.672019 6247 webserver.cc:460] Webserver started at http://127.28.94.2:34821/ using document root <none> and password file <none>
I20260806 04:29:50.672664 6247 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.672752 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.677342 6247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:50.679849 6264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.681320 6247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:50.681459 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "c4ef7401105f413db2ec467b71b49acc"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "fc0809449ea7c71437b960fc8733988c"
server_key_iv: "73bdade389abcaa22adf6a0f229bef60"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.682099 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.707741 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:50.711102 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.707744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.28.94.60:38615)
0806 04:29:50.707901 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.707906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.707925 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:29:50.707983 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.707986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.708044 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.708127 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:50.708390 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.708975 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.710017 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.710498 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.710602 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.710785 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.710868 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.710956 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:29:50.711102 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.707615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.0.0.1:43842)
0806 04:29:50.707753 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.707770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.707880 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.708156 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.708159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.708179 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:50.708278 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.708284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.709093 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.709097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.709869 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.709881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.710642 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.710646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.710747 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.710772 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.710840 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.710947 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:29:50.711580 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:50.711822 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:50.711946 5982 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 04:29:50.712358 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c2e2b9d21dc48498dc684f2b1fa7010. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:50.712518 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: LEARNER
I20260806 04:29:50.712604 6247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:50.713672 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:50.713727 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.713785 6247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:50.713814 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.723039 6247 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42151
I20260806 04:29:50.723068 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42151 every 8 connection(s)
I20260806 04:29:50.724236 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:50.726320 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 6247
I20260806 04:29:50.726521 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 5309
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
I20260806 04:29:50.755434 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.731329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:51303 (local address 127.28.94.61:35759)
0806 04:29:50.731454 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.731457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.733023 (+ 1566us) server_negotiation.cc:407] Connection header received
0806 04:29:50.734424 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.734431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.734510 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.734579 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.737322 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.738342 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.740709 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.741187 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.741298 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.746166 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.746173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.746177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.746221 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.748816 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.749690 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.749696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.749698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.749817 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.752957 (+ 3140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.752961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.752963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.753247 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.753340 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.755119 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.755265 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:29:50.755434 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.726557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:36235 (local address 127.28.94.62:36675)
0806 04:29:50.729942 (+ 3385us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.729947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.742852 (+ 12905us) server_negotiation.cc:407] Connection header received
0806 04:29:50.743330 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.743333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.743387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.743434 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.744143 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.744992 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.746034 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.746335 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.746420 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.749523 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.749528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.749531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.749566 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.751284 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.752513 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.752517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.752519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.752617 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.754117 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.754121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.754123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.754355 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.754429 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.754810 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.755271 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3243}
I20260806 04:29:50.757673 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.726926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.28.94.2:51303)
0806 04:29:50.732784 (+ 5858us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.732868 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.734014 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.735244 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.735259 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.736002 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.737069 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.737100 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.738561 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.738566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.740506 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.740522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.741597 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.741604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.741897 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.742838 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.742869 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.745834 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.749044 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.749052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.749076 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.749518 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.751518 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.751521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.751523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.751809 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.753422 (+ 1613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.753431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.753842 (+ 411us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.755665 (+ 1823us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.756489 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5547,"mutex_wait_us":391,"thread_start_us":2762,"threads_started":1}
I20260806 04:29:50.757680 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.726926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:36235)
0806 04:29:50.742751 (+ 15825us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.742768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.743224 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.743580 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.743582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.743815 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.744008 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.744018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.745192 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.745195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.745913 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.745921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.746589 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.746592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.746766 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.747536 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.747550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.749347 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.751904 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.751907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.751909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.752398 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.752724 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.752727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.752729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.752812 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.754501 (+ 1689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.754503 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.754593 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.755748 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.756405 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15704,"mutex_wait_us":5642,"thread_start_us":2514,"threads_started":1}
I20260806 04:29:50.758978 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.735685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:37785 (local address 127.28.94.60:38615)
0806 04:29:50.739737 (+ 4052us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.739741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.746087 (+ 6346us) server_negotiation.cc:407] Connection header received
0806 04:29:50.746155 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.746158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.746222 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.746299 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.747498 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.748341 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.751143 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.751507 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.751586 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.754611 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.754615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.754618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.754687 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.756826 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.757411 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.757415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.757416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.757514 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.757846 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.757850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.757852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.758143 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.758248 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.758690 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.758824 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3936}
I20260806 04:29:50.759408 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:33719
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:36675
--webserver_port=42111
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:29:50.759465 6429 heartbeater.cc:347] Connected to a master server at 127.28.94.61:35759
I20260806 04:29:50.759521 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.735809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.28.94.2:37785)
0806 04:29:50.743028 (+ 7219us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.743039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.743141 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.747038 (+ 3897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.747042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.747236 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.747362 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.747369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.749990 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.749993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.750915 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.750924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.751856 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.751860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.751964 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.752806 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.752820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.754433 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.756994 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.756998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.757000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.757280 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.757622 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.757625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.757627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.757733 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.758401 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.758404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.758506 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.759195 (+ 689us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.759355 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7160,"thread_start_us":441,"threads_started":1}
I20260806 04:29:50.759712 6429 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.760224 6429 heartbeater.cc:511] Master 127.28.94.61:35759 requested a full tablet report, sending...
I20260806 04:29:50.761415 6427 heartbeater.cc:347] Connected to a master server at 127.28.94.60:38615
I20260806 04:29:50.761511 6427 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.761498 5691 ts_manager.cc:194] Registered new tserver with Master: c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:50.761710 6427 heartbeater.cc:511] Master 127.28.94.60:38615 requested a full tablet report, sending...
I20260806 04:29:50.761833 6428 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:50.761898 6428 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.762055 6428 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:50.762506 5966 ts_manager.cc:194] Registered new tserver with Master: c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:50.762789 4863 ts_manager.cc:194] Re-registered known tserver with Master: c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:50.763423 4863 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:36235
W20260806 04:29:50.884995 6434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:50.885346 6434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:50.885412 6434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:50.890259 6434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:50.890352 6434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:50.890441 6434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:50.896454 6434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:33719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=42111
--tserver_master_addrs=127.28.94.62:36675,127.28.94.61:35759,127.28.94.60:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:50.897778 6434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:50.898962 6434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:50.909922 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:50.909919 6440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.910065 6434 server_base.cc:1034] running on GCE node
W20260806 04:29:50.910283 6439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:50.910763 6434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:50.911515 6434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:50.912711 6434 hybrid_clock.cc:648] HybridClock initialized: now 1785990590912675 us; error 55 us; skew 500 ppm
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:50.915784 6434 init.cc:377] Logged in from keytab as oryx/127.28.94.3@KRBTEST.COM (short username oryx)
I20260806 04:29:50.916867 6434 webserver.cc:460] Webserver started at http://127.28.94.3:42111/ using document root <none> and password file <none>
I20260806 04:29:50.917565 6434 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:50.917659 6434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:50.922026 6434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:29:50.924532 6451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.925997 6434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:50.926138 6434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "7988062833dc429f91442184b0969fc2"
format_stamp: "Formatted at 2026-08-06 04:29:49 on dist-test-slave-ncp9"
server_key: "e818444d40e47a0b549f7274344fafbc"
server_key_iv: "b3619b52bcc62227f6779ab840107b56"
server_key_version: "encryptionkey@0"
I20260806 04:29:50.926767 6434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:50.947897 6434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:50.951278 6434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:50.951483 6434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:50.952158 6434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:29:50.953167 6434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:50.953225 6434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.953285 6434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:50.953313 6434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:50.963585 6434 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:33719
I20260806 04:29:50.963610 6618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:33719 every 8 connection(s)
I20260806 04:29:50.964824 6434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:50.969547 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 6434
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.60@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.61@KRBTEST.COM
Aug 06 04:29:50 dist-test-slave-ncp9 krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990590, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:50.983160 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.969155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:34055 (local address 127.28.94.60:38615)
0806 04:29:50.969296 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.969301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.969386 (+ 85us) server_negotiation.cc:407] Connection header received
0806 04:29:50.970215 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.970220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.970281 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.970334 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.972644 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973371 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975107 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.975399 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975503 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.978873 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.978877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.978879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.978914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.980484 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.981290 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.981294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.981296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.981383 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.981761 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.981764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.981766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.981986 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.982119 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.982831 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.982972 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:29:50.984427 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.967292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:41425 (local address 127.28.94.62:36675)
0806 04:29:50.967432 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.967437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.968800 (+ 1363us) server_negotiation.cc:407] Connection header received
0806 04:29:50.970219 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.970223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.970282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.970334 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.973090 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973852 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975497 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.975830 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.980811 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.980815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.980818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.980853 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.982486 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.982992 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.982996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.982998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.983097 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.983418 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.983422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.983423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.983665 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.983753 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.984128 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.984268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":768}
I20260806 04:29:50.985929 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.967721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:41425)
0806 04:29:50.968606 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.968674 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.969945 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.970801 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.970811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.972777 (+ 1966us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.972943 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.972952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973998 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.974001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.975375 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.975387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.976056 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.976059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.976171 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.976721 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.976735 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.980674 (+ 3939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.982619 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.982622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.982625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.982868 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.983217 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.983220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.983222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.983312 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.983820 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.983824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.983932 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.984268 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.984689 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":423,"thread_start_us":146,"threads_started":1}
I20260806 04:29:50.987542 6623 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36675
I20260806 04:29:50.987826 6623 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.988442 6623 heartbeater.cc:511] Master 127.28.94.62:36675 requested a full tablet report, sending...
I20260806 04:29:50.988617 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.971683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.28.94.62:36675)
0806 04:29:50.974274 (+ 2591us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.974278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.974291 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:29:50.974327 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.974330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.974376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.975407 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.975512 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.976942 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.978190 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.978558 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.982405 (+ 3847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.982549 (+ 144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.982553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.982556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.982574 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.984771 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.985320 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.985323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.985324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.985480 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.985888 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.985891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.985893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.986124 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.987140 (+ 1016us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.987220 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.987351 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2505,"spinlock_wait_cycles":3968,"thread_start_us":82,"threads_started":1}
I20260806 04:29:50.989703 4861 ts_manager.cc:194] Re-registered known tserver with Master: 7988062833dc429f91442184b0969fc2 (127.28.94.3:33719)
I20260806 04:29:50.989961 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.967117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:33395 (local address 127.28.94.61:35759)
0806 04:29:50.967432 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.967437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.968800 (+ 1363us) server_negotiation.cc:407] Connection header received
0806 04:29:50.970219 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.970223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.970282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.970334 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.972642 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973449 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975690 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.976006 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.976088 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.979788 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.979793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.979796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.979832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.981499 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.987504 (+ 6005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.987508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.987510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.987618 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.987939 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.987942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.987944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.988175 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.988253 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.989674 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.989817 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260806 04:29:50.990481 4861 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:41425
I20260806 04:29:50.991851 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.967721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.28.94.3:33395)
0806 04:29:50.968606 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.968674 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.969945 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.970808 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.970812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.971552 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.972440 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.972466 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973590 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.973593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.975556 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.975567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.976280 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.976283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.976349 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.976863 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.976874 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.979599 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.987111 (+ 7512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.987114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.987116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.987402 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.987730 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.987733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.987735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.987828 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.988313 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.988316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.988413 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.991398 (+ 2985us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.991601 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":612,"spinlock_wait_cycles":73344,"thread_start_us":255,"threads_started":1}
I20260806 04:29:50.992509 6624 heartbeater.cc:347] Connected to a master server at 127.28.94.61:35759
I20260806 04:29:50.992616 6624 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.992787 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.968995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.28.94.3:34055)
0806 04:29:50.969291 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:50.969306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:50.969956 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:50.970801 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:50.970811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:50.971557 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:50.972441 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.972465 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.973548 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.973550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.974980 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:50.974990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.975692 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.975696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:50.975897 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.976585 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:50.976610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:50.978652 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:50.980786 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.980793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.980811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.981169 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.981486 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:50.981489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:50.981491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:50.981666 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:50.982213 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:50.982221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:50.982551 (+ 330us) client_negotiation.cc:769] Sending connection context
0806 04:29:50.992147 (+ 9596us) client_negotiation.cc:240] Negotiation successful
0806 04:29:50.992306 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":169,"spinlock_wait_cycles":5888,"thread_start_us":90,"threads_started":1}
I20260806 04:29:50.992794 6624 heartbeater.cc:511] Master 127.28.94.61:35759 requested a full tablet report, sending...
I20260806 04:29:50.993920 5691 ts_manager.cc:194] Registered new tserver with Master: 7988062833dc429f91442184b0969fc2 (127.28.94.3:33719)
I20260806 04:29:50.994441 6620 heartbeater.cc:347] Connected to a master server at 127.28.94.60:38615
I20260806 04:29:50.994527 6620 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:50.994735 6620 heartbeater.cc:511] Master 127.28.94.60:38615 requested a full tablet report, sending...
I20260806 04:29:50.996222 5966 ts_manager.cc:194] Registered new tserver with Master: 7988062833dc429f91442184b0969fc2 (127.28.94.3:33719)
I20260806 04:29:50.997238 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.972509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.28.94.61:35759)
0806 04:29:50.972788 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.972793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.976716 (+ 3923us) server_negotiation.cc:407] Connection header received
0806 04:29:50.976762 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.976765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.976819 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.976921 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.980585 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.986967 (+ 6382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.988498 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.988830 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.988954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.991107 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.991112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.991115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.991141 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.993732 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.994866 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.994870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.994872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.994972 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.995847 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.995851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.995853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.996087 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.996213 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.996926 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.997066 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20260806 04:29:50.994885 4861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:50.999491 4861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:50.999981 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:50.972146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.28.94.60:38615)
0806 04:29:50.983341 (+ 11195us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:50.983345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:50.983358 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:29:50.983392 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:50.983395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:50.983439 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:50.983508 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:50.984351 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.985101 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.994004 (+ 8903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:50.994333 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:50.994442 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:50.996460 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:50.996464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:50.996467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:50.996493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:50.998125 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.998601 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.998605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.998606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.998711 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:50.998954 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:50.998957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:50.998959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:50.999203 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:50.999347 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:50.999712 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 04:29:50.999827 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11134,"thread_start_us":99,"threads_started":1}
I20260806 04:29:51.002785 6643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: NON_VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: true } } }
I20260806 04:29:51.003044 6643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:51.006680 6642 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: attempting to promote NON_VOTER 296192dc335149b2a6e28c61fcb2f71d to VOTER
I20260806 04:29:51.007090 6642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:51.008457 5713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 04:29:51.008879 5912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:29:51.009487 5982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 LEARNER]: Refusing update from remote peer 5c2e2b9d21dc48498dc684f2b1fa7010: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 04:29:51.009888 5912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:29:51.011008 5912 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.012647 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.012763 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:51.012885 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.013020 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:51.013514 5713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.013757 6643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.013865 6643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010 [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:51.014387 5982 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.015341 6658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.015456 6658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:51.017221 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:51.017356 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:51.024356 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.019481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46015 (local address 127.0.0.1:54158)
0806 04:29:51.019625 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:51.019642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:51.019729 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:51.020720 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:51.020724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:51.020743 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:51.020862 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.020870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.021850 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.021853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.022790 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.022799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023933 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.023937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.024028 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.024051 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:29:51.024127 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:29:51.024205 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260806 04:29:51.024386 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.019804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.28.94.1:46015)
0806 04:29:51.020222 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.020230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.020257 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:29:51.020343 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.020349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.020576 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.020743 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:51.020979 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.021723 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.022934 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023790 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.023933 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.024114 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.024170 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.024269 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":73,"threads_started":1}
I20260806 04:29:51.026396 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.020165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.28.94.2:42151)
0806 04:29:51.021301 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.021313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.021340 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:29:51.021423 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.021428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.021647 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.021937 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:51.022082 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023292 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.024880 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.025760 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.026008 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.026140 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.026185 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.026264 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":120,"threads_started":1}
I20260806 04:29:51.026398 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.019968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42151 (local address 127.0.0.1:34308)
0806 04:29:51.020586 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:51.020602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:51.020698 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:51.021810 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:51.021813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:51.021843 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:51.021995 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.022002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023438 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.023442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.024712 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.024725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.025909 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.025913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.026034 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.026062 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:29:51.026140 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 04:29:51.026220 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":50,"spinlock_wait_cycles":1280,"thread_start_us":123,"threads_started":1}
I20260806 04:29:51.027534 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.020401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:33719 (local address 127.0.0.1:46542)
0806 04:29:51.020881 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:51.020954 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:51.021037 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:51.021802 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:51.021805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:51.021838 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:51.022157 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.022162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023481 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.023484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.024746 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.024753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.025893 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.025897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.026866 (+ 969us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.026892 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:29:51.027302 (+ 410us) client_negotiation.cc:240] Negotiation successful
0806 04:29:51.027389 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":175,"spinlock_wait_cycles":896,"thread_start_us":91,"threads_started":1}
I20260806 04:29:51.028061 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.020807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.28.94.3:33719)
0806 04:29:51.021325 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.021422 (+ 97us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.021443 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:29:51.021507 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.021511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.021653 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.021812 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:51.022283 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.023370 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.024871 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.025761 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.025877 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.026040 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.027030 (+ 990us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.027153 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":120,"threads_started":1}
I20260806 04:29:51.028100 6136 tablet_service.cc:1467] Processing CreateTablet for tablet c0d77197dc164ef9af1e0012bf43869d (DEFAULT_TABLE table=test-table [id=20e238cef744427f944b8a46b79ede36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:51.029394 6136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d77197dc164ef9af1e0012bf43869d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:51.029887 6329 tablet_service.cc:1467] Processing CreateTablet for tablet c0d77197dc164ef9af1e0012bf43869d (DEFAULT_TABLE table=test-table [id=20e238cef744427f944b8a46b79ede36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:51.030862 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d77197dc164ef9af1e0012bf43869d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:51.031117 6532 tablet_service.cc:1467] Processing CreateTablet for tablet c0d77197dc164ef9af1e0012bf43869d (DEFAULT_TABLE table=test-table [id=20e238cef744427f944b8a46b79ede36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:51.032205 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d77197dc164ef9af1e0012bf43869d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:51.039063 6668 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Bootstrap starting.
I20260806 04:29:51.039569 6666 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Bootstrap starting.
I20260806 04:29:51.041043 6670 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Bootstrap starting.
I20260806 04:29:51.041931 6668 tablet_bootstrap.cc:654] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:51.042102 6666 tablet_bootstrap.cc:654] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:51.043241 6670 tablet_bootstrap.cc:654] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:51.043269 6666 log.cc:824] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:51.043382 6668 log.cc:824] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:51.044224 6670 log.cc:824] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:51.045746 6666 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: No bootstrap required, opened a new log
I20260806 04:29:51.045768 6668 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: No bootstrap required, opened a new log
I20260806 04:29:51.045972 6668 ts_tablet_manager.cc:1397] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:29:51.045972 6666 ts_tablet_manager.cc:1397] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:29:51.046324 6670 tablet_bootstrap.cc:492] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: No bootstrap required, opened a new log
I20260806 04:29:51.046504 6670 ts_tablet_manager.cc:1397] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:51.050500 6670 raft_consensus.cc:348] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.050789 6670 raft_consensus.cc:374] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:51.050845 6670 raft_consensus.cc:729] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7988062833dc429f91442184b0969fc2, State: Initialized, Role: FOLLOWER
I20260806 04:29:51.050653 6668 raft_consensus.cc:348] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.050937 6668 raft_consensus.cc:374] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:51.050987 6668 raft_consensus.cc:729] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581de6cb02f54a2d9dcc8a6fef115468, State: Initialized, Role: FOLLOWER
I20260806 04:29:51.051144 6666 raft_consensus.cc:348] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.051416 6666 raft_consensus.cc:374] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:51.051350 6670 consensus_queue.cc:260] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.051476 6666 raft_consensus.cc:729] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ef7401105f413db2ec467b71b49acc, State: Initialized, Role: FOLLOWER
I20260806 04:29:51.051538 6668 consensus_queue.cc:260] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.051888 6666 consensus_queue.cc:260] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.052284 6623 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
I20260806 04:29:51.052671 6670 ts_tablet_manager.cc:1428] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:51.052697 6668 ts_tablet_manager.cc:1428] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:29:51.052873 6666 ts_tablet_manager.cc:1428] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:29:51.055469 6428 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
I20260806 04:29:51.055469 6235 heartbeater.cc:503] Master 127.28.94.62:36675 was elected leader, sending a full tablet report...
W20260806 04:29:51.216692 6625 tablet.cc:2369] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:51.225788 6430 tablet.cc:2369] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:51.235359 6237 tablet.cc:2369] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:51.265007 6675 raft_consensus.cc:482] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:51.265245 6675 raft_consensus.cc:504] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.266494 6675 leader_election.cc:288] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015), c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:51.270453 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.266897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:59599 (local address 127.28.94.2:42151)
0806 04:29:51.267017 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.267021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.267086 (+ 65us) server_negotiation.cc:407] Connection header received
0806 04:29:51.267234 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.267237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.267285 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.267347 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:51.267653 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.268262 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.269439 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.269890 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.269987 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.270060 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.270241 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.270318 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:51.270471 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.266817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42151 (local address 127.28.94.3:59599)
0806 04:29:51.267013 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:51.267027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:51.267120 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:51.267365 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:51.267368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:51.267400 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:51.267539 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.267545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.268367 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.268371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.269323 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.269334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.270016 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.270020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.270122 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.270150 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:29:51.270214 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:29:51.270297 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":18,"spinlock_wait_cycles":3328}
I20260806 04:29:51.270484 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.267016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37637 (local address 127.28.94.1:46015)
0806 04:29:51.267165 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.267168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.267180 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:29:51.267226 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.267228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.267269 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.267335 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:51.267656 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.268291 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.269438 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.269927 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.270027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.270096 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.270284 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.270358 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:29:51.270486 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.266745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46015 (local address 127.28.94.3:37637)
0806 04:29:51.266881 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:51.266899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:51.267006 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:51.267370 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:51.267373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:51.267406 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:51.267541 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.267546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.268396 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.268400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.269320 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:51.269331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.270041 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.270045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:51.270175 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.270193 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 04:29:51.270259 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:29:51.270335 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":19,"spinlock_wait_cycles":6144}
I20260806 04:29:51.271207 6362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d77197dc164ef9af1e0012bf43869d" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef7401105f413db2ec467b71b49acc" is_pre_election: true
I20260806 04:29:51.271215 6165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d77197dc164ef9af1e0012bf43869d" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581de6cb02f54a2d9dcc8a6fef115468" is_pre_election: true
I20260806 04:29:51.271528 6362 raft_consensus.cc:2454] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 0.
I20260806 04:29:51.271528 6165 raft_consensus.cc:2454] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 0.
I20260806 04:29:51.271996 6456 leader_election.cc:302] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7988062833dc429f91442184b0969fc2, c4ef7401105f413db2ec467b71b49acc; no voters:
I20260806 04:29:51.272300 6675 raft_consensus.cc:2781] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:51.272388 6675 raft_consensus.cc:482] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:51.272421 6675 raft_consensus.cc:3037] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:51.273711 6675 raft_consensus.cc:504] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.274135 6675 leader_election.cc:288] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 election: Requested vote from peers 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015), c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:51.274569 6165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d77197dc164ef9af1e0012bf43869d" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581de6cb02f54a2d9dcc8a6fef115468"
I20260806 04:29:51.274577 6362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d77197dc164ef9af1e0012bf43869d" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef7401105f413db2ec467b71b49acc"
I20260806 04:29:51.274717 6362 raft_consensus.cc:3037] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:51.274717 6165 raft_consensus.cc:3037] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:51.276067 6362 raft_consensus.cc:2454] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 1.
I20260806 04:29:51.276077 6165 raft_consensus.cc:2454] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 1.
I20260806 04:29:51.276437 6454 leader_election.cc:302] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 581de6cb02f54a2d9dcc8a6fef115468, 7988062833dc429f91442184b0969fc2, c4ef7401105f413db2ec467b71b49acc; no voters:
I20260806 04:29:51.276661 6675 raft_consensus.cc:2781] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:51.276917 6675 raft_consensus.cc:686] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 7988062833dc429f91442184b0969fc2, State: Running, Role: LEADER
I20260806 04:29:51.277339 6675 consensus_queue.cc:237] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } }
I20260806 04:29:51.281769 4861 catalog_manager.cc:5495] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7988062833dc429f91442184b0969fc2 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "7988062833dc429f91442184b0969fc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:51.320544 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.316704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.28.94.3:33719)
0806 04:29:51.316851 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.316856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.316871 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:51.317064 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.317067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.317143 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.317256 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:51.317504 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.318019 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.318877 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.319104 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.319201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.319526 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:51.319998 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:51.320122 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.320180 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.320251 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:29:51.327435 6165 raft_consensus.cc:1261] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Refusing update from remote peer 7988062833dc429f91442184b0969fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:51.327435 6362 raft_consensus.cc:1261] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Refusing update from remote peer 7988062833dc429f91442184b0969fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:51.328383 6683 consensus_queue.cc:1035] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:51.328606 6675 consensus_queue.cc:1035] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:51.338788 6688 mvcc.cc:205] Tried to move back new op lower bound from 7315417462068195328 to 7315417461874774016. Current Snapshot: MvccSnapshot[applied={T|T < 7315417462068195328}]
I20260806 04:29:51.341264 6689 mvcc.cc:205] Tried to move back new op lower bound from 7315417462068195328 to 7315417461874774016. Current Snapshot: MvccSnapshot[applied={T|T < 7315417462068195328}]
I20260806 04:29:51.350072 4860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:51.350273 4860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:51.355326 4860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c2e2b9d21dc48498dc684f2b1fa7010: Sending DeleteTablet for 3 replicas of tablet c0d77197dc164ef9af1e0012bf43869d
I20260806 04:29:51.356280 6136 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d77197dc164ef9af1e0012bf43869d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:51 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:54158
I20260806 04:29:51.356562 6329 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d77197dc164ef9af1e0012bf43869d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:51 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:34308
I20260806 04:29:51.356752 6532 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d77197dc164ef9af1e0012bf43869d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:51 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46542
I20260806 04:29:51.357108 6701 tablet_replica.cc:330] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: stopping tablet replica
I20260806 04:29:51.358098 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 4822
I20260806 04:29:51.358135 6701 raft_consensus.cc:2229] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:51.363229 6701 raft_consensus.cc:2258] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:51.365742 6701 ts_tablet_manager.cc:1905] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:51.368269 6701 ts_tablet_manager.cc:1918] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:51.368391 6701 log.cc:1196] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/c0d77197dc164ef9af1e0012bf43869d
I20260806 04:29:51.368710 6701 ts_tablet_manager.cc:1939] T c0d77197dc164ef9af1e0012bf43869d P 581de6cb02f54a2d9dcc8a6fef115468: Deleting consensus metadata
I20260806 04:29:51.369213 6702 tablet_replica.cc:330] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: stopping tablet replica
I20260806 04:29:51.369227 6705 tablet_replica.cc:330] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: stopping tablet replica
I20260806 04:29:51.369632 6705 raft_consensus.cc:2229] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:51.369663 6702 raft_consensus.cc:2229] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:51.370138 6705 raft_consensus.cc:2258] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:51.370261 6702 raft_consensus.cc:2258] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:51.372381 6702 ts_tablet_manager.cc:1905] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:51.375252 6702 ts_tablet_manager.cc:1918] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:51.375367 6702 log.cc:1196] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/c0d77197dc164ef9af1e0012bf43869d
I20260806 04:29:51.375645 6702 ts_tablet_manager.cc:1939] T c0d77197dc164ef9af1e0012bf43869d P c4ef7401105f413db2ec467b71b49acc: Deleting consensus metadata
W20260806 04:29:51.376140 6068 connection.cc:535] server connection from 127.0.0.1:54158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 04:29:51.376976 6268 connection.cc:441] server connection from 127.0.0.1:34308 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34308 (request call id 1) could send its response
I20260806 04:29:51.382648 6705 ts_tablet_manager.cc:1905] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:51.386024 6705 ts_tablet_manager.cc:1918] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:51.386178 6705 log.cc:1196] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/c0d77197dc164ef9af1e0012bf43869d
I20260806 04:29:51.386569 6705 ts_tablet_manager.cc:1939] T c0d77197dc164ef9af1e0012bf43869d P 7988062833dc429f91442184b0969fc2: Deleting consensus metadata
W20260806 04:29:51.387972 6452 connection.cc:441] server connection from 127.0.0.1:46542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46542 (request call id 1) could send its response
I20260806 04:29:51.390637 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.358254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.28.94.61:35759)
0806 04:29:51.361312 (+ 3058us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.361318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.361340 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:29:51.361400 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.361403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.361473 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.361589 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:51.369230 (+ 7641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.370023 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.371348 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.371699 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.371784 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.386506 (+ 14722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:51.386511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:51.386514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:51.386547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:51.388520 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.389136 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.389141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.389142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.389242 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.389568 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.389571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.389573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.389826 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:51.389940 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.390309 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.390452 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2927}
I20260806 04:29:51.390838 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.381133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.28.94.60:38615)
0806 04:29:51.381273 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.381278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.381293 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:29:51.381335 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.381338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.381386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.381463 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:51.382541 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.383302 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.384478 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.384812 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.384936 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.386985 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:51.386989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:51.386992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:51.387022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:51.388803 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.389432 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.389436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.389437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.389533 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.389840 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.389843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.389845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.390069 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:51.390181 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.390525 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.390674 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:51.392017 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.381448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.28.94.61:35759)
0806 04:29:51.381566 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.381570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.382662 (+ 1092us) server_negotiation.cc:407] Connection header received
0806 04:29:51.382818 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.382821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.382869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.382916 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:51.383904 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.384712 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.386158 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.386480 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.386548 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.388274 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:51.388278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:51.388281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:51.388311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:51.389889 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.390435 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.390438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.390440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.390515 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.390816 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.390819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.390821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.391051 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:51.391157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.391762 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.391872 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:51.403834 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:51.358360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.28.94.60:38615)
0806 04:29:51.362680 (+ 4320us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:51.362685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:51.392447 (+ 29762us) server_negotiation.cc:407] Connection header received
0806 04:29:51.393138 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:51.393141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:51.393195 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:51.393248 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:51.394164 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.394774 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.395568 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:51.395937 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:51.396052 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:51.399225 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:51.399230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:51.399233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:51.399263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:51.400977 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.401460 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.401464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.401466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.401564 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:51.402133 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:51.402136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:51.402138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:51.402649 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:51.402842 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:51.403435 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 04:29:51.403640 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4198}
I20260806 04:29:52.342545 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.341716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:52353)
0806 04:29:52.342313 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.342423 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":86,"threads_started":1}
I20260806 04:29:52.342857 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.342143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:54523)
0806 04:29:52.342632 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.342754 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":83,"threads_started":1}
W20260806 04:29:52.343129 6623 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:36675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
W20260806 04:29:52.343398 6235 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:36675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.343596 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.342827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:32909)
0806 04:29:52.343394 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.343497 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":92,"threads_started":1}
W20260806 04:29:52.343878 6428 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:36675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.855188 6722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c2e2b9d21dc48498dc684f2b1fa7010)
I20260806 04:29:52.855443 6722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:52.856312 6723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c2e2b9d21dc48498dc684f2b1fa7010)
I20260806 04:29:52.856491 6723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:52.856647 6722 leader_election.cc:288] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60:38615)
I20260806 04:29:52.857275 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.856801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56610)
0806 04:29:52.857071 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.857161 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":55,"threads_started":1}
I20260806 04:29:52.857527 6723 leader_election.cc:288] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61:35759)
W20260806 04:29:52.857888 5676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.857995 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.857569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56626)
0806 04:29:52.857814 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.857901 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
W20260806 04:29:52.858374 5941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.859212 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.858978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56642)
0806 04:29:52.859071 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.859126 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 04:29:52.859364 5676 leader_election.cc:334] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.860252 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.860012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56654)
0806 04:29:52.860131 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.860184 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 04:29:52.860458 5941 leader_election.cc:334] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.862316 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.857732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:35759 (local address 127.0.0.1:43954)
0806 04:29:52.858678 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.858693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:52.858837 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:52.859203 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:52.859206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:52.859223 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:52.859328 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:52.859335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.860274 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.860276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:52.861251 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:52.861262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.861895 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.861899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:52.861984 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:52.862004 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:29:52.862068 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:29:52.862148 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":57,"thread_start_us":62,"threads_started":1}
I20260806 04:29:52.862326 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.857788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.28.94.61:35759)
0806 04:29:52.858227 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:52.858232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:52.858759 (+ 527us) server_negotiation.cc:407] Connection header received
0806 04:29:52.858954 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:52.858958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:52.859002 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:52.859058 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:52.859444 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.860161 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.861369 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.861777 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.861871 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:52.861936 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:52.862093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 04:29:52.862174 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":55,"threads_started":1}
I20260806 04:29:52.862718 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.857163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:38615 (local address 127.0.0.1:43888)
0806 04:29:52.857426 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.857439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:52.857550 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:52.859680 (+ 2130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:52.859683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:52.859702 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:52.859798 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:52.859805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.860801 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.860805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:52.861659 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:52.861669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.862336 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.862340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:52.862427 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:52.862452 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:29:52.862523 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:29:52.862597 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260806 04:29:52.862835 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "296192dc335149b2a6e28c61fcb2f71d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" is_pre_election: true
I20260806 04:29:52.862996 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.857408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.28.94.60:38615)
0806 04:29:52.858892 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:52.858902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:52.858916 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:29:52.858962 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:52.858965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:52.859540 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:52.859936 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:52.860016 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.860697 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.861783 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:52.862225 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:52.862316 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:52.862372 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:52.862778 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 04:29:52.862865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":1323,"threads_started":1}
I20260806 04:29:52.863080 5713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296192dc335149b2a6e28c61fcb2f71d in term 1.
I20260806 04:29:52.863394 5982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "296192dc335149b2a6e28c61fcb2f71d" is_pre_election: true
I20260806 04:29:52.863458 5945 leader_election.cc:302] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00ff06d82d2d4901a399cd8fd9d3c85e, 296192dc335149b2a6e28c61fcb2f71d; no voters: 5c2e2b9d21dc48498dc684f2b1fa7010
I20260806 04:29:52.863590 5982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00ff06d82d2d4901a399cd8fd9d3c85e in term 1.
I20260806 04:29:52.863693 6723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:29:52.863752 6723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5c2e2b9d21dc48498dc684f2b1fa7010)
I20260806 04:29:52.863798 6723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:29:52.863935 5676 leader_election.cc:302] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00ff06d82d2d4901a399cd8fd9d3c85e, 296192dc335149b2a6e28c61fcb2f71d; no voters: 5c2e2b9d21dc48498dc684f2b1fa7010
I20260806 04:29:52.864135 6722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:29:52.864197 6722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5c2e2b9d21dc48498dc684f2b1fa7010)
I20260806 04:29:52.864244 6722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:29:52.865007 6723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:52.865403 6723 leader_election.cc:288] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 election: Requested vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61:35759)
I20260806 04:29:52.865443 6722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:52.865824 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "296192dc335149b2a6e28c61fcb2f71d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
I20260806 04:29:52.865844 6722 leader_election.cc:288] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 election: Requested vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 296192dc335149b2a6e28c61fcb2f71d (127.28.94.60:38615)
I20260806 04:29:52.866019 5713 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 296192dc335149b2a6e28c61fcb2f71d in current term 2: Already voted for candidate 00ff06d82d2d4901a399cd8fd9d3c85e in this term.
I20260806 04:29:52.866061 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.865555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56662)
0806 04:29:52.865931 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.865993 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260806 04:29:52.866348 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.865967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56668)
0806 04:29:52.866227 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.866281 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260806 04:29:52.866317 5982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "296192dc335149b2a6e28c61fcb2f71d"
I20260806 04:29:52.866468 5982 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 00ff06d82d2d4901a399cd8fd9d3c85e in current term 2: Already voted for candidate 296192dc335149b2a6e28c61fcb2f71d in this term.
I20260806 04:29:52.866899 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.866509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56670)
0806 04:29:52.866790 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.866839 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":197}
W20260806 04:29:52.866974 5941 leader_election.cc:334] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.867028 5941 leader_election.cc:302] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 296192dc335149b2a6e28c61fcb2f71d; no voters: 00ff06d82d2d4901a399cd8fd9d3c85e, 5c2e2b9d21dc48498dc684f2b1fa7010
I20260806 04:29:52.867115 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:52.866906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56684)
0806 04:29:52.867009 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:52.867057 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 04:29:52.867179 6723 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 04:29:52.867314 5676 leader_election.cc:334] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:52.867367 5676 leader_election.cc:302] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00ff06d82d2d4901a399cd8fd9d3c85e; no voters: 296192dc335149b2a6e28c61fcb2f71d, 5c2e2b9d21dc48498dc684f2b1fa7010
I20260806 04:29:52.867482 6722 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:29:53.344252 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:53.343723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:47755)
0806 04:29:53.344066 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:53.344135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":86,"threads_started":1}
I20260806 04:29:53.344538 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:53.343970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:38845)
0806 04:29:53.344333 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:53.344430 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":89,"threads_started":1}
I20260806 04:29:53.344882 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:53.344365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:59735)
0806 04:29:53.344729 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:53.344790 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":90,"threads_started":1}
I20260806 04:29:54.345682 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.345000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:42667)
0806 04:29:54.345462 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.345533 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":95,"threads_started":1}
I20260806 04:29:54.345834 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.345398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:45521)
0806 04:29:54.345679 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.345748 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260806 04:29:54.346237 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.345537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:59495)
0806 04:29:54.346081 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.346143 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":61,"threads_started":1}
I20260806 04:29:54.678671 6741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:54.678833 6741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:54.679354 6741 leader_election.cc:288] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61:35759)
I20260806 04:29:54.679853 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "296192dc335149b2a6e28c61fcb2f71d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" is_pre_election: true
I20260806 04:29:54.679945 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.679562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56878)
0806 04:29:54.679793 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.679855 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260806 04:29:54.680012 5713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 296192dc335149b2a6e28c61fcb2f71d in term 2.
I20260806 04:29:54.680351 5945 leader_election.cc:302] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ff06d82d2d4901a399cd8fd9d3c85e, 296192dc335149b2a6e28c61fcb2f71d; no voters:
I20260806 04:29:54.680539 6741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:29:54.680596 6741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:54.680635 6741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:29:54.680872 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.680645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56892)
0806 04:29:54.680752 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.680807 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 04:29:54.681012 5941 leader_election.cc:334] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:54.681816 6741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:54.682138 6741 leader_election.cc:288] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 election: Requested vote from peers 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675), 00ff06d82d2d4901a399cd8fd9d3c85e (127.28.94.61:35759)
I20260806 04:29:54.682469 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.682289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56894)
0806 04:29:54.682374 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.682414 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 04:29:54.682605 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "296192dc335149b2a6e28c61fcb2f71d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e"
I20260806 04:29:54.682751 5713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:29:54.683126 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.682918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56896)
0806 04:29:54.683017 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.683069 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:29:54.683266 5941 leader_election.cc:334] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
I20260806 04:29:54.683858 5713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 296192dc335149b2a6e28c61fcb2f71d in term 3.
I20260806 04:29:54.684135 5945 leader_election.cc:302] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00ff06d82d2d4901a399cd8fd9d3c85e, 296192dc335149b2a6e28c61fcb2f71d; no voters: 5c2e2b9d21dc48498dc684f2b1fa7010
I20260806 04:29:54.684310 6741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:29:54.684518 6741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [term 3 LEADER]: Becoming Leader. State: Replica: 296192dc335149b2a6e28c61fcb2f71d, State: Running, Role: LEADER
I20260806 04:29:54.684791 6741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } }
I20260806 04:29:54.686280 6744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 296192dc335149b2a6e28c61fcb2f71d. Latest consensus state: current_term: 3 leader_uuid: "296192dc335149b2a6e28c61fcb2f71d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:54.686384 6744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:54.686677 6746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:54.689114 6746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:54.689502 6746 catalog_manager.cc:1259] Loaded cluster ID: 5a202b740b8b44aebd774d9c936bbf58
I20260806 04:29:54.689545 6746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:54.690069 6746 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:54.690474 6746 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Loaded TSK: 0
I20260806 04:29:54.691161 6746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:29:54.695547 5966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:54.698881 5966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:54.702350 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.702072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56914)
0806 04:29:54.702213 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.702266 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:29:54.702447 5713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [term 3 FOLLOWER]: Refusing update from remote peer 296192dc335149b2a6e28c61fcb2f71d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 04:29:54.703042 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.702830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.0.0.1:56928)
0806 04:29:54.702933 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.702980 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:29:54.703009 6744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 04:29:54.703174 5941 consensus_peers.cc:487] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d -> Peer 5c2e2b9d21dc48498dc684f2b1fa7010 (127.28.94.62:36675): Couldn't send request to peer 5c2e2b9d21dc48498dc684f2b1fa7010. Status: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:29:54.704725 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 296192dc335149b2a6e28c61fcb2f71d. Latest consensus state: current_term: 3 leader_uuid: "296192dc335149b2a6e28c61fcb2f71d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:54.704859 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:54.706085 6741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "296192dc335149b2a6e28c61fcb2f71d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:54.706221 6741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:54.706604 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "296192dc335149b2a6e28c61fcb2f71d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5c2e2b9d21dc48498dc684f2b1fa7010" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36675 } } peers { permanent_uuid: "00ff06d82d2d4901a399cd8fd9d3c85e" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 35759 } attrs { promote: false } } peers { permanent_uuid: "296192dc335149b2a6e28c61fcb2f71d" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 38615 } attrs { promote: false } } }
I20260806 04:29:54.706714 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ff06d82d2d4901a399cd8fd9d3c85e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:29:54.717734 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.713187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46015 (local address 127.0.0.1:54160)
0806 04:29:54.713457 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.713470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:54.713560 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:54.714074 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:54.714077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:54.714097 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:54.714217 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.714225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715217 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.715220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.716381 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.716392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.717346 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.717349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.717422 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.717441 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:29:54.717503 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 04:29:54.717557 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260806 04:29:54.717746 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.713568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.28.94.1:46015)
0806 04:29:54.713812 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:54.713816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:54.713835 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:29:54.713902 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:54.713905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:54.713964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:54.714043 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:54.714325 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715079 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.716525 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.717221 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.717349 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.717420 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:54.717519 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 04:29:54.717598 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260806 04:29:54.717825 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.712996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.28.94.3:33719)
0806 04:29:54.713833 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:54.713849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:54.713862 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:29:54.713907 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:54.713910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:54.713954 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:54.714044 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:54.714565 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715297 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.716349 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.716987 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.717090 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.717167 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:54.717628 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 04:29:54.717707 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":101,"threads_started":1}
I20260806 04:29:54.718116 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.712857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:33719 (local address 127.0.0.1:46564)
0806 04:29:54.713065 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.713081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:54.713180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:54.714343 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:54.714346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:54.714363 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:54.714446 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.714452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715395 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.715398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.716210 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.716219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.717262 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.717266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.717365 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.717391 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:29:54.717464 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:29:54.717545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":24}
I20260806 04:29:54.718397 6136 tablet_service.cc:1467] Processing CreateTablet for tablet 27fde136abd8424fba78383ce937f424 (DEFAULT_TABLE table=test-table [id=d83edbe2174b4912bfeb5359feedbe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:54.718644 6532 tablet_service.cc:1467] Processing CreateTablet for tablet 27fde136abd8424fba78383ce937f424 (DEFAULT_TABLE table=test-table [id=d83edbe2174b4912bfeb5359feedbe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:54.718724 6136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fde136abd8424fba78383ce937f424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:54.718955 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fde136abd8424fba78383ce937f424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:54.719178 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.713602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42151 (local address 127.0.0.1:34324)
0806 04:29:54.714474 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.714486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:54.714579 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:54.714821 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:54.714824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:54.714840 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:54.714945 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.714950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715944 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.715948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.717114 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:54.717126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.718757 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.718761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:54.718855 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.718881 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:29:54.718958 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:29:54.719040 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260806 04:29:54.719250 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.713951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.28.94.2:42151)
0806 04:29:54.714352 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:54.714356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:54.714641 (+ 285us) server_negotiation.cc:407] Connection header received
0806 04:29:54.714691 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:54.714693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:54.714731 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:54.714798 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:54.715051 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.715813 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.717257 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.718519 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.718623 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.718692 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:54.719015 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 04:29:54.719122 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":88,"threads_started":1}
I20260806 04:29:54.719815 6329 tablet_service.cc:1467] Processing CreateTablet for tablet 27fde136abd8424fba78383ce937f424 (DEFAULT_TABLE table=test-table [id=d83edbe2174b4912bfeb5359feedbe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:54.720089 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fde136abd8424fba78383ce937f424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:54.721571 6765 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: Bootstrap starting.
I20260806 04:29:54.722647 6765 tablet_bootstrap.cc:654] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:54.722805 6767 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Bootstrap starting.
I20260806 04:29:54.723642 6766 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Bootstrap starting.
I20260806 04:29:54.723759 6767 tablet_bootstrap.cc:654] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:54.725131 6767 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: No bootstrap required, opened a new log
I20260806 04:29:54.725131 6765 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: No bootstrap required, opened a new log
I20260806 04:29:54.725236 6765 ts_tablet_manager.cc:1397] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 04:29:54.725308 6767 ts_tablet_manager.cc:1397] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:54.725836 6766 tablet_bootstrap.cc:654] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:54.725818 6767 raft_consensus.cc:348] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.725970 6767 raft_consensus.cc:374] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:54.725818 6765 raft_consensus.cc:348] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.726046 6767 raft_consensus.cc:729] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ef7401105f413db2ec467b71b49acc, State: Initialized, Role: FOLLOWER
I20260806 04:29:54.726125 6765 raft_consensus.cc:374] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:54.726215 6767 consensus_queue.cc:260] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.726261 6765 raft_consensus.cc:729] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581de6cb02f54a2d9dcc8a6fef115468, State: Initialized, Role: FOLLOWER
I20260806 04:29:54.726488 6765 consensus_queue.cc:260] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.726658 6767 ts_tablet_manager.cc:1428] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:54.727644 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.727254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:51181)
0806 04:29:54.727499 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.727564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 04:29:54.729626 6427 heartbeater.cc:503] Master 127.28.94.60:38615 was elected leader, sending a full tablet report...
I20260806 04:29:54.729997 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.729741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:45713)
0806 04:29:54.729861 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.729919 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 04:29:54.730964 6765 ts_tablet_manager.cc:1428] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260806 04:29:54.731796 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.731424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:36467)
0806 04:29:54.731654 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.731715 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260806 04:29:54.732298 6766 tablet_bootstrap.cc:492] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: No bootstrap required, opened a new log
I20260806 04:29:54.732419 6766 ts_tablet_manager.cc:1397] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260806 04:29:54.732996 6234 heartbeater.cc:503] Master 127.28.94.60:38615 was elected leader, sending a full tablet report...
I20260806 04:29:54.733027 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.732501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:34907)
0806 04:29:54.732853 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.732946 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20260806 04:29:54.732951 6766 raft_consensus.cc:348] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.733070 6766 raft_consensus.cc:374] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:54.733103 6766 raft_consensus.cc:729] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7988062833dc429f91442184b0969fc2, State: Initialized, Role: FOLLOWER
I20260806 04:29:54.733224 6766 consensus_queue.cc:260] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.733659 6766 ts_tablet_manager.cc:1428] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:54.734329 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.733955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:52437)
0806 04:29:54.734199 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.734258 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 04:29:54.735113 6620 heartbeater.cc:503] Master 127.28.94.60:38615 was elected leader, sending a full tablet report...
I20260806 04:29:54.735457 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.735159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:59887)
0806 04:29:54.735344 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.735397 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 04:29:54.770946 6774 raft_consensus.cc:482] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:54.771124 6774 raft_consensus.cc:504] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.771597 6774 leader_election.cc:288] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015), c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:54.772141 6165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fde136abd8424fba78383ce937f424" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581de6cb02f54a2d9dcc8a6fef115468" is_pre_election: true
I20260806 04:29:54.772154 6362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fde136abd8424fba78383ce937f424" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef7401105f413db2ec467b71b49acc" is_pre_election: true
I20260806 04:29:54.772295 6165 raft_consensus.cc:2454] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 0.
I20260806 04:29:54.772320 6362 raft_consensus.cc:2454] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 0.
I20260806 04:29:54.772666 6454 leader_election.cc:302] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 581de6cb02f54a2d9dcc8a6fef115468, 7988062833dc429f91442184b0969fc2, c4ef7401105f413db2ec467b71b49acc; no voters:
I20260806 04:29:54.772854 6774 raft_consensus.cc:2781] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:54.772949 6774 raft_consensus.cc:482] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:54.772996 6774 raft_consensus.cc:3037] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:54.774075 6774 raft_consensus.cc:504] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.774441 6774 leader_election.cc:288] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 election: Requested vote from peers 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015), c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151)
I20260806 04:29:54.774812 6362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fde136abd8424fba78383ce937f424" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4ef7401105f413db2ec467b71b49acc"
I20260806 04:29:54.774814 6165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fde136abd8424fba78383ce937f424" candidate_uuid: "7988062833dc429f91442184b0969fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581de6cb02f54a2d9dcc8a6fef115468"
I20260806 04:29:54.774950 6362 raft_consensus.cc:3037] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:54.774955 6165 raft_consensus.cc:3037] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:54.775877 6165 raft_consensus.cc:2454] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 1.
I20260806 04:29:54.775888 6362 raft_consensus.cc:2454] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7988062833dc429f91442184b0969fc2 in term 1.
I20260806 04:29:54.776167 6456 leader_election.cc:302] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7988062833dc429f91442184b0969fc2, c4ef7401105f413db2ec467b71b49acc; no voters:
I20260806 04:29:54.776386 6774 raft_consensus.cc:2781] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:54.776597 6774 raft_consensus.cc:686] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 7988062833dc429f91442184b0969fc2, State: Running, Role: LEADER
I20260806 04:29:54.776750 6774 consensus_queue.cc:237] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } }
I20260806 04:29:54.777359 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.777117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:39839)
0806 04:29:54.777228 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.777288 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 04:29:54.778514 5965 catalog_manager.cc:5495] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7988062833dc429f91442184b0969fc2 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "7988062833dc429f91442184b0969fc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7988062833dc429f91442184b0969fc2" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 33719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:54.798735 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.795894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.28.94.3:33719)
0806 04:29:54.796035 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:54.796039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:54.796053 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:29:54.796104 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:54.796106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:54.796154 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:54.796215 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:54.796396 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.796948 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.797805 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:54.798070 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:54.798148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:54.798330 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:54.798404 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:54.798448 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:54.798537 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 04:29:54.798590 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:29:54.801184 6362 raft_consensus.cc:1261] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Refusing update from remote peer 7988062833dc429f91442184b0969fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:54.801184 6165 raft_consensus.cc:1261] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Refusing update from remote peer 7988062833dc429f91442184b0969fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:54.801663 6774 consensus_queue.cc:1035] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "581de6cb02f54a2d9dcc8a6fef115468" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:54.801854 6776 consensus_queue.cc:1035] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4ef7401105f413db2ec467b71b49acc" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:54.803457 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.803147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:56077)
0806 04:29:54.803310 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.803371 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 04:29:54.804479 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.804049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:50559)
0806 04:29:54.804342 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.804401 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260806 04:29:54.806421 6781 mvcc.cc:205] Tried to move back new op lower bound from 7315417476301045760 to 7315417476207153152. Current Snapshot: MvccSnapshot[applied={T|T < 7315417476301045760}]
I20260806 04:29:54.808266 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.807909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.1:59391)
0806 04:29:54.808116 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.808177 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 04:29:54.809149 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.807264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:46021)
0806 04:29:54.809010 (+ 1746us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.809071 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1665}
I20260806 04:29:54.810504 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.810198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.2:37703)
0806 04:29:54.810364 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.810423 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 04:29:54.812836 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.812573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36675 (local address 127.28.94.3:58233)
0806 04:29:54.812693 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.812754 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:29:54.813751 5965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:54.813912 5965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:54.817792 5965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 296192dc335149b2a6e28c61fcb2f71d: Sending DeleteTablet for 3 replicas of tablet 27fde136abd8424fba78383ce937f424
I20260806 04:29:54.818531 6532 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fde136abd8424fba78383ce937f424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:54 UTC) from {username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:46564
I20260806 04:29:54.818655 6136 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fde136abd8424fba78383ce937f424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:54 UTC) from {username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:54160
I20260806 04:29:54.818650 6329 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fde136abd8424fba78383ce937f424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:54 UTC) from {username='oryx', principal='oryx/127.28.94.60@KRBTEST.COM'} at 127.0.0.1:34324
I20260806 04:29:54.818946 6793 tablet_replica.cc:330] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: stopping tablet replica
I20260806 04:29:54.818975 6794 tablet_replica.cc:330] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468: stopping tablet replica
I20260806 04:29:54.819092 6794 raft_consensus.cc:2229] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:54.819092 6793 raft_consensus.cc:2229] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:54.819232 6794 raft_consensus.cc:2258] T 27fde136abd8424fba78383ce937f424 P 581de6cb02f54a2d9dcc8a6fef115468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:54.819300 6793 raft_consensus.cc:2258] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:54.819423 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 5907
I20260806 04:29:54.820021 6795 tablet_replica.cc:330] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: stopping tablet replica
I20260806 04:29:54.820143 6795 raft_consensus.cc:2229] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:54.820258 6795 raft_consensus.cc:2258] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:54.822103 6793 ts_tablet_manager.cc:1905] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:54.823628 6795 ts_tablet_manager.cc:1905] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:54.824996 6793 ts_tablet_manager.cc:1918] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:54.825088 6793 log.cc:1196] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/27fde136abd8424fba78383ce937f424
I20260806 04:29:54.825377 6793 ts_tablet_manager.cc:1939] T 27fde136abd8424fba78383ce937f424 P 7988062833dc429f91442184b0969fc2: Deleting consensus metadata
I20260806 04:29:54.826184 6795 ts_tablet_manager.cc:1918] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:54.826253 6795 log.cc:1196] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/27fde136abd8424fba78383ce937f424
I20260806 04:29:54.826459 6795 ts_tablet_manager.cc:1939] T 27fde136abd8424fba78383ce937f424 P c4ef7401105f413db2ec467b71b49acc: Deleting consensus metadata
I20260806 04:29:54.826741 5945 catalog_manager.cc:4841] TS 7988062833dc429f91442184b0969fc2 (127.28.94.3:33719): tablet 27fde136abd8424fba78383ce937f424 (table test-table [id=d83edbe2174b4912bfeb5359feedbe19]) successfully deleted
I20260806 04:29:54.827275 5945 catalog_manager.cc:4841] TS c4ef7401105f413db2ec467b71b49acc (127.28.94.2:42151): tablet 27fde136abd8424fba78383ce937f424 (table test-table [id=d83edbe2174b4912bfeb5359feedbe19]) successfully deleted
I20260806 04:29:54.842214 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 6247
I20260806 04:29:54.846901 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:54.842353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46015 (local address 127.0.0.1:54162)
0806 04:29:54.846702 (+ 4349us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:54.846791 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:46015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4215}
W20260806 04:29:54.850270 5942 catalog_manager.cc:4584] TS 581de6cb02f54a2d9dcc8a6fef115468 (127.28.94.1:46015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27fde136abd8424fba78383ce937f424: Network error: Client connection negotiation failed: client connection to 127.28.94.1:46015: connect: Connection refused (error 111)
I20260806 04:29:54.860261 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 6434
I20260806 04:29:54.875684 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 5504
I20260806 04:29:54.888547 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 5661
2026-08-06T04:29:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7614 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:54 dist-test-slave-ncp9 krb5kdc[6800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:54 dist-test-slave-ncp9 krb5kdc[6800](info): set up 2 sockets
Aug 06 04:29:54 dist-test-slave-ncp9 krb5kdc[6800](info): commencing operation
krb5kdc: starting...
W20260806 04:29:56.011793 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:29:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab.
Entry for principal oryx/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab.
I20260806 04:29:56.143904 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:32777
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab
--principal=oryx/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:56.267567 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:56.267903 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:56.267962 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:56.272271 6819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:29:56.272360 6819 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:29:56.272384 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:56.272409 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:56.272430 6819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:29:56.272450 6819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:29:56.278488 6819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:32777
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.62.keytab
--principal=oryx/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:56.279758 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:56.280949 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:56.290393 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:56.290494 6819 server_base.cc:1034] running on GCE node
W20260806 04:29:56.290391 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:56.290391 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:56.291373 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:56.292626 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:56.293792 6819 hybrid_clock.cc:648] HybridClock initialized: now 1785990596293764 us; error 40 us; skew 500 ppm
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:56.296614 6819 init.cc:377] Logged in from keytab as oryx/127.28.94.62@KRBTEST.COM (short username oryx)
I20260806 04:29:56.297670 6819 webserver.cc:460] Webserver started at http://127.28.94.62:43189/ using document root <none> and password file <none>
I20260806 04:29:56.298280 6819 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:56.298353 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:56.298576 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:56.300784 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "9ac4e4788d6946ccbedbd4ea387fed9f"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "917806a43ac6880d170d955af4ba464a"
server_key_iv: "b3686706c7c7129b756ec3cc68db53a4"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.301414 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "9ac4e4788d6946ccbedbd4ea387fed9f"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "917806a43ac6880d170d955af4ba464a"
server_key_iv: "b3686706c7c7129b756ec3cc68db53a4"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.305447 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:29:56.308169 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:56.309610 6819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:29:56.309752 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "9ac4e4788d6946ccbedbd4ea387fed9f"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "917806a43ac6880d170d955af4ba464a"
server_key_iv: "b3686706c7c7129b756ec3cc68db53a4"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.309870 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:56.332140 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:56.335244 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:56.335449 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:56.343973 6819 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:32777
I20260806 04:29:56.343986 6917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:32777 every 8 connection(s)
I20260806 04:29:56.345032 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:29:56.347996 6918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:56.352578 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 6819
I20260806 04:29:56.352741 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:29:56.353091 29048 external_mini_cluster.cc:1428] Setting key bb522c8e10eca2273d27bf70de906c60
I20260806 04:29:56.354645 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Bootstrap starting.
I20260806 04:29:56.357314 6918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:56.358155 6918 log.cc:824] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:56.360282 6918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: No bootstrap required, opened a new log
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:56.364962 6918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } }
I20260806 04:29:56.365334 6918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:56.365398 6918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac4e4788d6946ccbedbd4ea387fed9f, State: Initialized, Role: FOLLOWER
I20260806 04:29:56.365962 6918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } }
I20260806 04:29:56.366163 6918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:29:56.366243 6918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:29:56.366338 6918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:56.367671 6918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } }
I20260806 04:29:56.368098 6918 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac4e4788d6946ccbedbd4ea387fed9f; no voters:
I20260806 04:29:56.368475 6918 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:29:56.368630 6924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:56.368860 6924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [term 1 LEADER]: Becoming Leader. State: Replica: 9ac4e4788d6946ccbedbd4ea387fed9f, State: Running, Role: LEADER
I20260806 04:29:56.369047 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.354792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.28.94.62:32777)
0806 04:29:56.355334 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:56.355350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:56.355392 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:29:56.356118 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:56.356137 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:56.356519 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:56.356874 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:56.357777 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.358836 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.359694 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.360189 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.360342 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.363352 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:56.363360 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:56.363384 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:56.363426 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:56.365436 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.365905 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.365911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.365917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.366080 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.366490 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.366493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.366495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.367085 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:56.367314 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:56.367521 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 04:29:56.367890 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":157,"threads_started":1}
I20260806 04:29:56.369270 6924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } }
I20260806 04:29:56.369956 6918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:29:56.371111 6926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac4e4788d6946ccbedbd4ea387fed9f. Latest consensus state: current_term: 1 leader_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } } }
I20260806 04:29:56.371250 6926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:56.371589 6925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac4e4788d6946ccbedbd4ea387fed9f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 32777 } } }
I20260806 04:29:56.371683 6925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f [sys.catalog]: This master's current role is: LEADER
I20260806 04:29:56.372575 6929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:29:56.375770 6929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:29:56.383441 6929 catalog_manager.cc:1347] Generated new cluster ID: 26f493f0bd7d4595bad8066151178015
I20260806 04:29:56.383543 6929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:29:56.414286 6929 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:29:56.415681 6929 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:29:56.420984 6929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Generated new TSK 0
I20260806 04:29:56.421916 6929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
I20260806 04:29:56.454269 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:32777
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:56.579197 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:56.579543 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:56.579596 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:56.583954 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:56.584053 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:56.584100 6952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:56.584149 6952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:29:56.590247 6952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:56.591529 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:56.592674 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:56.603099 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:56.603091 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:56.603091 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:56.603246 6952 server_base.cc:1034] running on GCE node
I20260806 04:29:56.603857 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:56.604625 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:56.605782 6952 hybrid_clock.cc:648] HybridClock initialized: now 1785990596605735 us; error 65 us; skew 500 ppm
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:56.611297 6952 init.cc:377] Logged in from keytab as oryx/127.28.94.1@KRBTEST.COM (short username oryx)
I20260806 04:29:56.612401 6952 webserver.cc:460] Webserver started at http://127.28.94.1:43789/ using document root <none> and password file <none>
I20260806 04:29:56.613049 6952 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:56.613122 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:56.613348 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:56.615633 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "2a6e46e995a34df08c8e2cd45ab44208"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "724fb1896b6a69f88d40075322741d05"
server_key_iv: "08e8ef57cd799f24b2e73aa05df6c0df"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.616215 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "2a6e46e995a34df08c8e2cd45ab44208"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "724fb1896b6a69f88d40075322741d05"
server_key_iv: "08e8ef57cd799f24b2e73aa05df6c0df"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.620422 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:56.623378 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:56.624876 6952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:56.625059 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "2a6e46e995a34df08c8e2cd45ab44208"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "724fb1896b6a69f88d40075322741d05"
server_key_iv: "08e8ef57cd799f24b2e73aa05df6c0df"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.625183 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:56.666214 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:56.669791 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:56.670007 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:56.671767 6952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:56.671826 6952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:56.671885 6952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:56.671913 6952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:56.682843 6952 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:34079
I20260806 04:29:56.682964 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:34079 every 8 connection(s)
I20260806 04:29:56.684186 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:29:56.686584 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.672744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:52559 (local address 127.28.94.62:32777)
0806 04:29:56.672926 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:56.672931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:56.674160 (+ 1229us) server_negotiation.cc:407] Connection header received
0806 04:29:56.675381 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:56.675386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:56.675458 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:56.675563 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:56.677141 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.677752 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.679069 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.679325 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.679423 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.682420 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:56.682425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:56.682428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:56.682465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:56.684197 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.684863 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.684867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.684869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.685020 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.685432 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.685436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.685437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.685686 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:56.685772 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:56.686270 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 04:29:56.686395 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:29:56.687459 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.673424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.1:52559)
0806 04:29:56.673998 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:56.674047 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:56.675086 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:56.675757 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:56.675768 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:56.676193 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:56.676965 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.676987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.677887 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.677890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.678932 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.678943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.679516 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.679520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.679752 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.680385 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:56.680403 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:56.682173 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:56.684364 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.684370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.684384 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.684713 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.685126 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.685129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.685131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.685325 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.685773 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:56.685778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:56.686038 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 04:29:56.686213 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 04:29:56.686507 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":135,"threads_started":1}
I20260806 04:29:56.688979 7129 heartbeater.cc:347] Connected to a master server at 127.28.94.62:32777
I20260806 04:29:56.689270 7129 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:56.689936 7129 heartbeater.cc:511] Master 127.28.94.62:32777 requested a full tablet report, sending...
I20260806 04:29:56.691866 6860 ts_manager.cc:194] Registered new tserver with Master: 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079)
I20260806 04:29:56.693223 6860 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:52559
I20260806 04:29:56.693282 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 6952
I20260806 04:29:56.693410 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:29:56.693903 29048 external_mini_cluster.cc:1428] Setting key 58659ba3414043d2a76a2d79085e372f
I20260806 04:29:56.700671 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.691952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:56215 (local address 127.28.94.62:32777)
0806 04:29:56.692085 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:56.692090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:56.693242 (+ 1152us) server_negotiation.cc:407] Connection header received
0806 04:29:56.693394 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:56.693398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:56.693569 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:56.693684 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:56.694227 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.694947 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.695932 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.696164 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.696254 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.697475 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:56.697479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:56.697482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:56.697506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:56.698968 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.699444 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.699448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.699450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.699533 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.699809 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.699812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.699814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.700021 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:56.700119 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:56.700381 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 04:29:56.700501 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:29:56.700696 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.691820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.1:56215)
0806 04:29:56.693128 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:56.693145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:56.693270 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:56.693718 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:56.693721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:56.693962 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:56.694099 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.694107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.695099 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.695103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.695808 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.695820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.696274 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.696278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.696374 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.696935 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:56.696952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:56.697323 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:56.699117 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.699120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.699121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.699339 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.699631 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.699634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.699635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.699720 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.700118 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:56.700121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:56.700223 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 04:29:56.700312 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:29:56.700487 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for oryx/127.28.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
I20260806 04:29:56.724004 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:32777
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:56.844959 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:56.845269 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:56.845317 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:56.849486 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:56.849563 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:56.849610 7141 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:56.849643 7141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:29:56.855331 7141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:56.856573 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:56.857723 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:56.868157 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:56.868162 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:56.868325 7141 server_base.cc:1034] running on GCE node
W20260806 04:29:56.868252 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:56.868943 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:56.869683 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:56.870857 7141 hybrid_clock.cc:648] HybridClock initialized: now 1785990596870830 us; error 36 us; skew 500 ppm
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:56.874153 7141 init.cc:377] Logged in from keytab as oryx/127.28.94.2@KRBTEST.COM (short username oryx)
I20260806 04:29:56.875216 7141 webserver.cc:460] Webserver started at http://127.28.94.2:37013/ using document root <none> and password file <none>
I20260806 04:29:56.875823 7141 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:56.875891 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:56.876078 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:56.878486 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "ecfcef129e604c1fa39f75f801a2ddec"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "1557831d784b5510b22dd39ad31fa5bc"
server_key_iv: "f272695328baccb2830dca1f8f0eb011"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.879082 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "ecfcef129e604c1fa39f75f801a2ddec"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "1557831d784b5510b22dd39ad31fa5bc"
server_key_iv: "f272695328baccb2830dca1f8f0eb011"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.883404 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 04:29:56.886173 7161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:56.887526 7141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:56.887670 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "ecfcef129e604c1fa39f75f801a2ddec"
format_stamp: "Formatted at 2026-08-06 04:29:56 on dist-test-slave-ncp9"
server_key: "1557831d784b5510b22dd39ad31fa5bc"
server_key_iv: "f272695328baccb2830dca1f8f0eb011"
server_key_version: "encryptionkey@0"
I20260806 04:29:56.887795 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:56.898870 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:56.902277 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:56.902477 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:56.904086 7141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:56.904148 7141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:56.904204 7141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:56.904234 7141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:29:56 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:56.914862 7141 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43219
I20260806 04:29:56.914953 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43219 every 8 connection(s)
I20260806 04:29:56.916132 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:29:56.918861 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.905050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:59155 (local address 127.28.94.62:32777)
0806 04:29:56.905223 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:56.905228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:56.906161 (+ 933us) server_negotiation.cc:407] Connection header received
0806 04:29:56.907311 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:56.907315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:56.907383 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:56.907489 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:56.909078 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.909635 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.911051 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.911286 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.911391 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.914612 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:56.914618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:56.914621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:56.914658 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:56.916287 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.916989 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.916994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.916996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.917118 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.917523 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.917526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.917528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.917757 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:56.917861 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:56.918517 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 04:29:56.918672 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:29:56.919682 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.905434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.2:59155)
0806 04:29:56.905976 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:56.906034 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:56.907057 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:56.907725 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:56.907736 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:56.908230 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:56.908870 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.908887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.909777 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.909781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.910900 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.910913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.911481 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.911486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.911682 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.912482 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:56.912509 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:56.914341 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:56.916418 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.916424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.916438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.916801 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.917214 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.917218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.917221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.917376 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.917877 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:56.917885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:56.918225 (+ 340us) client_negotiation.cc:769] Sending connection context
0806 04:29:56.918415 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 04:29:56.918786 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260806 04:29:56.921062 7310 heartbeater.cc:347] Connected to a master server at 127.28.94.62:32777
I20260806 04:29:56.921229 7310 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:56.921794 7310 heartbeater.cc:511] Master 127.28.94.62:32777 requested a full tablet report, sending...
I20260806 04:29:56.922433 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 7141
I20260806 04:29:56.922593 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:29:56.922947 29048 external_mini_cluster.cc:1428] Setting key 3f7da93752617f3a9807f9b0f9358f96
I20260806 04:29:56.922940 6860 ts_manager.cc:194] Registered new tserver with Master: ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:56.923650 6860 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:59155
I20260806 04:29:56.931113 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.923517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46777 (local address 127.28.94.62:32777)
0806 04:29:56.923648 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:56.923653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:56.923924 (+ 271us) server_negotiation.cc:407] Connection header received
0806 04:29:56.924092 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:56.924095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:56.924146 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:56.924228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:56.924770 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.925532 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.926474 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.926703 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.926792 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.927974 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:56.927977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:56.927979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:56.928004 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:56.929365 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.929888 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.929891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.929893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.929975 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:56.930239 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:56.930243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:56.930245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:56.930477 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:56.930572 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:56.930838 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 04:29:56.930947 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:29:56.931124 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:56.923506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.2:46777)
0806 04:29:56.923844 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:56.923859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:56.923993 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:56.924259 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:56.924262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:56.924503 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:56.924634 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.924642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.925695 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.925699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.926357 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:56.926367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:56.926811 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:56.926815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:56.926918 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:56.927473 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:56.927485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:56.927832 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:56.929476 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.929479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.929482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.929722 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.930068 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:56.930071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:56.930073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:56.930157 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:56.930584 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:56.930587 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:56.930682 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 04:29:56.930769 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:29:56.930936 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
I20260806 04:29:56.954206 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:32777
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:29:57.077690 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:29:57.078001 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:29:57.078048 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:29:57.082320 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:29:57.082394 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:29:57.082437 7322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:29:57.082469 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:29:57.088239 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.94.62:32777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:29:57.089569 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:29:57.090715 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:29:57.101228 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:57.101426 7322 server_base.cc:1034] running on GCE node
W20260806 04:29:57.101183 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:29:57.101183 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:29:57.102061 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:29:57.102836 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:29:57.104014 7322 hybrid_clock.cc:648] HybridClock initialized: now 1785990597103994 us; error 36 us; skew 500 ppm
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990597, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:29:57.107023 7322 init.cc:377] Logged in from keytab as oryx/127.28.94.3@KRBTEST.COM (short username oryx)
I20260806 04:29:57.108048 7322 webserver.cc:460] Webserver started at http://127.28.94.3:45763/ using document root <none> and password file <none>
I20260806 04:29:57.108696 7322 fs_manager.cc:359] Metadata directory not provided
I20260806 04:29:57.108778 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:29:57.109063 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:29:57.111358 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "8bebb37b2d384532b5a92a6c67d88755"
format_stamp: "Formatted at 2026-08-06 04:29:57 on dist-test-slave-ncp9"
server_key: "3043bdae967cf24c9f00f31cdb338948"
server_key_iv: "d680b5ac8d78339a2ee44f6d1d4be5c2"
server_key_version: "encryptionkey@0"
I20260806 04:29:57.111981 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "8bebb37b2d384532b5a92a6c67d88755"
format_stamp: "Formatted at 2026-08-06 04:29:57 on dist-test-slave-ncp9"
server_key: "3043bdae967cf24c9f00f31cdb338948"
server_key_iv: "d680b5ac8d78339a2ee44f6d1d4be5c2"
server_key_version: "encryptionkey@0"
I20260806 04:29:57.116290 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:29:57.119242 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:57.120589 7322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:57.120733 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "8bebb37b2d384532b5a92a6c67d88755"
format_stamp: "Formatted at 2026-08-06 04:29:57 on dist-test-slave-ncp9"
server_key: "3043bdae967cf24c9f00f31cdb338948"
server_key_iv: "d680b5ac8d78339a2ee44f6d1d4be5c2"
server_key_version: "encryptionkey@0"
I20260806 04:29:57.120850 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:29:57.135293 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:29:57.138827 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:29:57.139046 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:29:57.140738 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:29:57.140802 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:29:57.140866 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:29:57.140933 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990597, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:57.151943 7322 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40499
I20260806 04:29:57.152045 7492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40499 every 8 connection(s)
I20260806 04:29:57.153010 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:29:57.155609 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.141716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:36763 (local address 127.28.94.62:32777)
0806 04:29:57.141893 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.141899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.142965 (+ 1066us) server_negotiation.cc:407] Connection header received
0806 04:29:57.144297 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.144301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.144367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.144453 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:57.145922 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.146464 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.148342 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.148694 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.148820 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.152022 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:57.152027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:57.152030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:57.152067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:57.153713 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.154251 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.154254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.154256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.154348 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.154627 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.154629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.154630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.154831 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:57.154912 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.155366 (+ 454us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.155476 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:29:57.156508 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.142138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.3:36763)
0806 04:29:57.142755 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.142824 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.143988 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.144718 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.144728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.145167 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:57.145749 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.145766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.146647 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.146651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.148195 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.148211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.148885 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.148913 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.149135 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.149822 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:57.149847 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:57.151762 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:57.153864 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.153870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.153889 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.154160 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.154421 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.154422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.154424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.154555 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.154909 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:57.154914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:57.155172 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.155331 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.155631 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":122,"threads_started":1}
I20260806 04:29:57.157863 7493 heartbeater.cc:347] Connected to a master server at 127.28.94.62:32777
I20260806 04:29:57.158020 7493 heartbeater.cc:464] Registering TS with master...
I20260806 04:29:57.158620 7493 heartbeater.cc:511] Master 127.28.94.62:32777 requested a full tablet report, sending...
I20260806 04:29:57.159777 6860 ts_manager.cc:194] Registered new tserver with Master: 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499)
I20260806 04:29:57.160588 6860 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:36763
I20260806 04:29:57.162861 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 7322
I20260806 04:29:57.162992 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:29:57.163362 29048 external_mini_cluster.cc:1428] Setting key 1a699784bc56d866b52ad936f119a362
I20260806 04:29:57.165825 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:29:57.168000 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.160192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.28.94.3:47599)
0806 04:29:57.160511 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.160524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.160625 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.160844 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.160848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.161101 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:57.161226 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.161235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.161998 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.162001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.162802 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.162813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.163370 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.163373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.163465 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.164097 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:57.164109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:57.164492 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:57.166550 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.166553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.166555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.166762 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.167040 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.167042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.167043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.167115 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.167525 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:57.167528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:57.167626 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.167708 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.167839 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":135,"threads_started":1}
I20260806 04:29:57.168064 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.160235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:47599 (local address 127.28.94.62:32777)
0806 04:29:57.160349 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.160354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.160594 (+ 240us) server_negotiation.cc:407] Connection header received
0806 04:29:57.160704 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.160707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.160750 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.160816 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:57.161352 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.161860 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.162942 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.163257 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.163370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.164634 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:57.164637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:57.164639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:57.164670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:57.166417 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.166861 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.166864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.166866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.166960 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.167190 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.167193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.167195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.167433 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:57.167529 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.167780 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.167912 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:29:57.193423 6921 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:57.186819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.28.94.62:32777)
0806 04:29:57.186967 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.186971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.186983 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:29:57.187027 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.187030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.187073 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.187137 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:57.187903 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.188477 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.189460 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.189692 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.189787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.193245 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56240: BlockingRecv error: recv got EOF from 127.0.0.1:56240 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:57.202646 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.194876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.28.94.62:32777)
0806 04:29:57.195031 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.195035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.195099 (+ 64us) server_negotiation.cc:407] Connection header received
0806 04:29:57.195200 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.195203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.195248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.195314 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:57.195904 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.196438 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.197292 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.197534 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.197616 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.199885 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:57.199889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:57.199891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:57.199921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:57.201248 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.201629 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.201632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.201633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.201709 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.201900 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.201901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.201902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.202085 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:57.202183 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.202371 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.202475 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:29:57.205703 6860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:57.208853 6860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:57.225512 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.220060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.28.94.2:43219)
0806 04:29:57.220495 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.220507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.220728 (+ 221us) server_negotiation.cc:407] Connection header received
0806 04:29:57.220824 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.220831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.221079 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.221235 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.221721 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.222591 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.223973 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.224723 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.224833 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.225101 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.225231 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.225343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":113,"threads_started":1}
I20260806 04:29:57.225521 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.219253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:43219 (local address 127.0.0.1:35918)
0806 04:29:57.220618 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.220629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.220714 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.221393 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.221396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.221415 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.221601 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.221610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.222731 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.222734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.223830 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.223841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.224849 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.224853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.225008 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.225062 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.225192 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.225341 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1312,"mutex_wait_us":31,"thread_start_us":108,"threads_started":1}
I20260806 04:29:57.226651 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.219581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40499 (local address 127.0.0.1:60828)
0806 04:29:57.220652 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.220670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.220763 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.221215 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.221222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.221244 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.221382 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.221391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.222841 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.222844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.224172 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.224183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.226260 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.226264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.226349 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.226373 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.226446 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.226526 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":45,"thread_start_us":95,"threads_started":1}
I20260806 04:29:57.226666 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.220061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.28.94.3:40499)
0806 04:29:57.220448 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.220491 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.220765 (+ 274us) server_negotiation.cc:407] Connection header received
0806 04:29:57.220850 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.220854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.221065 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.221213 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.221520 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.222706 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.224303 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.226138 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.226235 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.226370 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.226478 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.226554 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260806 04:29:57.226854 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.219192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34079 (local address 127.0.0.1:46358)
0806 04:29:57.220214 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.220274 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.220436 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.221424 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.221426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.221445 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.221631 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.221638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.223272 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.223275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.224418 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.224428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.225583 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.225586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.225672 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.225697 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.225757 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.225831 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":28,"thread_start_us":93,"threads_started":1}
I20260806 04:29:57.227579 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.219462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.28.94.1:34079)
0806 04:29:57.221028 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.221035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.221057 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:29:57.221125 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.221131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.221302 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.221438 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.221779 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.223142 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.224543 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.225289 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.227112 (+ 1823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.227286 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.227340 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.227445 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":104,"threads_started":1}
I20260806 04:29:57.229301 7224 tablet_service.cc:1467] Processing CreateTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 (DEFAULT_TABLE table=test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.230639 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088cf806ae2d4febbf54c8091999e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.230674 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 (DEFAULT_TABLE table=test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.231671 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088cf806ae2d4febbf54c8091999e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.232160 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 (DEFAULT_TABLE table=test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.233362 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088cf806ae2d4febbf54c8091999e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.238623 7528 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Bootstrap starting.
I20260806 04:29:57.240991 7528 tablet_bootstrap.cc:654] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.241844 7528 log.cc:824] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:57.242560 7530 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Bootstrap starting.
I20260806 04:29:57.242676 7529 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Bootstrap starting.
I20260806 04:29:57.244745 7529 tablet_bootstrap.cc:654] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.244748 7530 tablet_bootstrap.cc:654] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.245560 7530 log.cc:824] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:57.245639 7529 log.cc:824] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Log is configured to *not* fsync() on all Append() calls
I20260806 04:29:57.247133 7528 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: No bootstrap required, opened a new log
I20260806 04:29:57.247370 7528 ts_tablet_manager.cc:1397] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 04:29:57.247751 7530 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: No bootstrap required, opened a new log
I20260806 04:29:57.247750 7529 tablet_bootstrap.cc:492] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: No bootstrap required, opened a new log
I20260806 04:29:57.247953 7530 ts_tablet_manager.cc:1397] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:29:57.247953 7529 ts_tablet_manager.cc:1397] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:29:57.251394 7528 raft_consensus.cc:348] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.251677 7528 raft_consensus.cc:374] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.251758 7528 raft_consensus.cc:729] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6e46e995a34df08c8e2cd45ab44208, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.251998 7529 raft_consensus.cc:348] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.251998 7530 raft_consensus.cc:348] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.252274 7530 raft_consensus.cc:374] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.252274 7529 raft_consensus.cc:374] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.252334 7529 raft_consensus.cc:729] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.252260 7528 consensus_queue.cc:260] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.252367 7530 raft_consensus.cc:729] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.252801 7529 consensus_queue.cc:260] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.252801 7530 consensus_queue.cc:260] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.253373 7129 heartbeater.cc:503] Master 127.28.94.62:32777 was elected leader, sending a full tablet report...
I20260806 04:29:57.253392 7528 ts_tablet_manager.cc:1428] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:57.253858 7529 ts_tablet_manager.cc:1428] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:57.253858 7530 ts_tablet_manager.cc:1428] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:29:57.254238 7493 heartbeater.cc:503] Master 127.28.94.62:32777 was elected leader, sending a full tablet report...
I20260806 04:29:57.254602 7310 heartbeater.cc:503] Master 127.28.94.62:32777 was elected leader, sending a full tablet report...
W20260806 04:29:57.404222 7495 tablet.cc:2369] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:57.417613 7313 tablet.cc:2369] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:29:57.436141 7130 tablet.cc:2369] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:29:57.507881 7536 raft_consensus.cc:482] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:57.508117 7536 raft_consensus.cc:504] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.509440 7536 leader_election.cc:288] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499)
I20260806 04:29:57.513652 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.509771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34079 (local address 127.28.94.2:33455)
0806 04:29:57.510083 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.510098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.510226 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.510537 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.510540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.510589 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.510684 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.510689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.511419 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.511422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.512353 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.512364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.513091 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.513094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.513279 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.513313 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.513381 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.513477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":128,"spinlock_wait_cycles":896,"thread_start_us":81,"threads_started":1}
I20260806 04:29:57.513633 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.509898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:33455 (local address 127.28.94.1:34079)
0806 04:29:57.510058 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.510063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.510179 (+ 116us) server_negotiation.cc:407] Connection header received
0806 04:29:57.510360 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.510363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.510410 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.510483 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.510796 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.511311 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.512489 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.512965 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.513080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.513150 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.513413 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.513496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:29:57.513680 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.509898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:49253 (local address 127.28.94.3:40499)
0806 04:29:57.510097 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.510102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.510239 (+ 137us) server_negotiation.cc:407] Connection header received
0806 04:29:57.510373 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.510376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.510421 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.510486 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.510779 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.511311 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.512486 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.512887 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.513029 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.513102 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.513450 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.513537 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 04:29:57.514084 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.509774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40499 (local address 127.28.94.2:49253)
0806 04:29:57.510152 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.510165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.510270 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.510516 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.510519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.510536 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.510669 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.510674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.511425 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.511428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.512354 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.512365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.513067 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.513071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.513283 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.513320 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.513848 (+ 528us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.513942 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":121,"thread_start_us":65,"threads_started":1}
I20260806 04:29:57.514706 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088cf806ae2d4febbf54c8091999e4f8" candidate_uuid: "ecfcef129e604c1fa39f75f801a2ddec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bebb37b2d384532b5a92a6c67d88755" is_pre_election: true
I20260806 04:29:57.514703 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088cf806ae2d4febbf54c8091999e4f8" candidate_uuid: "ecfcef129e604c1fa39f75f801a2ddec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208" is_pre_election: true
I20260806 04:29:57.515027 7060 raft_consensus.cc:2454] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecfcef129e604c1fa39f75f801a2ddec in term 0.
I20260806 04:29:57.515028 7427 raft_consensus.cc:2454] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecfcef129e604c1fa39f75f801a2ddec in term 0.
I20260806 04:29:57.515547 7162 leader_election.cc:302] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, ecfcef129e604c1fa39f75f801a2ddec; no voters:
I20260806 04:29:57.515843 7536 raft_consensus.cc:2781] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:57.515941 7536 raft_consensus.cc:482] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:57.515993 7536 raft_consensus.cc:3037] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.517163 7536 raft_consensus.cc:504] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.517539 7536 leader_election.cc:288] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [CANDIDATE]: Term 1 election: Requested vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499)
I20260806 04:29:57.517993 7427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088cf806ae2d4febbf54c8091999e4f8" candidate_uuid: "ecfcef129e604c1fa39f75f801a2ddec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bebb37b2d384532b5a92a6c67d88755"
I20260806 04:29:57.517993 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088cf806ae2d4febbf54c8091999e4f8" candidate_uuid: "ecfcef129e604c1fa39f75f801a2ddec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208"
I20260806 04:29:57.518143 7060 raft_consensus.cc:3037] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.518146 7427 raft_consensus.cc:3037] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.519587 7060 raft_consensus.cc:2454] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecfcef129e604c1fa39f75f801a2ddec in term 1.
I20260806 04:29:57.519773 7427 raft_consensus.cc:2454] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecfcef129e604c1fa39f75f801a2ddec in term 1.
I20260806 04:29:57.520066 7162 leader_election.cc:302] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, ecfcef129e604c1fa39f75f801a2ddec; no voters:
I20260806 04:29:57.520280 7536 raft_consensus.cc:2781] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:57.520481 7536 raft_consensus.cc:686] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 1 LEADER]: Becoming Leader. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Running, Role: LEADER
I20260806 04:29:57.520854 7536 consensus_queue.cc:237] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.524588 6860 catalog_manager.cc:5495] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2). New cstate: current_term: 1 leader_uuid: "ecfcef129e604c1fa39f75f801a2ddec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } health_report { overall_health: UNKNOWN } } }
I20260806 04:29:57.598717 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.594997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.28.94.2:43219)
0806 04:29:57.595163 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.595167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.595184 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:29:57.595277 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.595279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.595339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.595438 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:57.595724 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.596244 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.597151 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.597378 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.597470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.597774 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.598304 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.598416 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.598494 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.598560 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:29:57.605396 7427 raft_consensus.cc:1261] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Refusing update from remote peer ecfcef129e604c1fa39f75f801a2ddec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:57.605396 7060 raft_consensus.cc:1261] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Refusing update from remote peer ecfcef129e604c1fa39f75f801a2ddec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:57.606204 7539 consensus_queue.cc:1035] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:57.606423 7536 consensus_queue.cc:1035] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:57.615651 7544 mvcc.cc:205] Tried to move back new op lower bound from 7315417487783575552 to 7315417487448657920. Current Snapshot: MvccSnapshot[applied={T|T < 7315417487783575552}]
I20260806 04:29:57.615875 7545 mvcc.cc:205] Tried to move back new op lower bound from 7315417487783575552 to 7315417487448657920. Current Snapshot: MvccSnapshot[applied={T|T < 7315417487783575552}]
I20260806 04:29:57.626030 6857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:57.626240 6857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:57.629254 6857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Sending DeleteTablet for 3 replicas of tablet 088cf806ae2d4febbf54c8091999e4f8
I20260806 04:29:57.630369 7224 tablet_service.cc:1514] Processing DeleteTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:57 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35918
I20260806 04:29:57.630519 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:57 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260806 04:29:57.631032 7558 tablet_replica.cc:330] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: stopping tablet replica
I20260806 04:29:57.631213 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 088cf806ae2d4febbf54c8091999e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:57 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260806 04:29:57.631434 7558 raft_consensus.cc:2229] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:57.631398 6857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:57.631901 6857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:57.631932 7558 raft_consensus.cc:2258] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:57.632004 7560 tablet_replica.cc:330] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: stopping tablet replica
I20260806 04:29:57.632381 7560 raft_consensus.cc:2229] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:57.632783 7560 raft_consensus.cc:2258] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:57.633639 7558 ts_tablet_manager.cc:1905] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:57.634563 7560 ts_tablet_manager.cc:1905] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:57.636613 7558 ts_tablet_manager.cc:1918] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:57.636734 7558 log.cc:1196] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/088cf806ae2d4febbf54c8091999e4f8
I20260806 04:29:57.637086 7558 ts_tablet_manager.cc:1939] T 088cf806ae2d4febbf54c8091999e4f8 P ecfcef129e604c1fa39f75f801a2ddec: Deleting consensus metadata
I20260806 04:29:57.637382 7557 tablet_replica.cc:330] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: stopping tablet replica
I20260806 04:29:57.637508 7560 ts_tablet_manager.cc:1918] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:57.637645 7560 log.cc:1196] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/088cf806ae2d4febbf54c8091999e4f8
I20260806 04:29:57.637806 7557 raft_consensus.cc:2229] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:57.637982 7560 ts_tablet_manager.cc:1939] T 088cf806ae2d4febbf54c8091999e4f8 P 8bebb37b2d384532b5a92a6c67d88755: Deleting consensus metadata
I20260806 04:29:57.638206 7557 raft_consensus.cc:2258] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:57.639014 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 1c06b217a4464adb987de39641e672f1 (DEFAULT_TABLE table=test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.639309 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c06b217a4464adb987de39641e672f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.639926 7557 ts_tablet_manager.cc:1905] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:57.641358 7530 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Bootstrap starting.
I20260806 04:29:57.641631 6841 catalog_manager.cc:4841] TS 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499): tablet 088cf806ae2d4febbf54c8091999e4f8 (table test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]) successfully deleted
I20260806 04:29:57.642436 7530 tablet_bootstrap.cc:654] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.642796 7557 ts_tablet_manager.cc:1918] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:57.642894 7557 log.cc:1196] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/088cf806ae2d4febbf54c8091999e4f8
I20260806 04:29:57.643225 7557 ts_tablet_manager.cc:1939] T 088cf806ae2d4febbf54c8091999e4f8 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting consensus metadata
I20260806 04:29:57.643399 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 1c06b217a4464adb987de39641e672f1 (DEFAULT_TABLE table=test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.643677 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c06b217a4464adb987de39641e672f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.642578 7224 tablet_service.cc:1467] Processing CreateTablet for tablet 1c06b217a4464adb987de39641e672f1 (DEFAULT_TABLE table=test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:57.644667 6838 catalog_manager.cc:4841] TS 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079): tablet 088cf806ae2d4febbf54c8091999e4f8 (table test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]) successfully deleted
I20260806 04:29:57.644774 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c06b217a4464adb987de39641e672f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.645830 7528 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Bootstrap starting.
I20260806 04:29:57.646875 7528 tablet_bootstrap.cc:654] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.646970 7530 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: No bootstrap required, opened a new log
I20260806 04:29:57.647078 7530 ts_tablet_manager.cc:1397] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 04:29:57.647291 7529 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Bootstrap starting.
I20260806 04:29:57.647575 7530 raft_consensus.cc:348] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.647696 7530 raft_consensus.cc:374] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.647727 7530 raft_consensus.cc:729] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.647866 7530 consensus_queue.cc:260] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.648134 7530 ts_tablet_manager.cc:1428] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:57.648336 7529 tablet_bootstrap.cc:654] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.649273 7528 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: No bootstrap required, opened a new log
I20260806 04:29:57.649255 6839 catalog_manager.cc:4841] TS ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219): tablet 088cf806ae2d4febbf54c8091999e4f8 (table test-table [id=bcda79ecc5294c4ca0aac12a330c0d66]) successfully deleted
I20260806 04:29:57.649379 7528 ts_tablet_manager.cc:1397] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 04:29:57.649684 7529 tablet_bootstrap.cc:492] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: No bootstrap required, opened a new log
I20260806 04:29:57.649768 7529 ts_tablet_manager.cc:1397] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:57.650121 7528 raft_consensus.cc:348] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.650213 7528 raft_consensus.cc:374] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.650233 7528 raft_consensus.cc:729] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6e46e995a34df08c8e2cd45ab44208, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.650264 7529 raft_consensus.cc:348] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.650378 7529 raft_consensus.cc:374] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.650345 7528 consensus_queue.cc:260] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.650413 7529 raft_consensus.cc:729] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.650557 7529 consensus_queue.cc:260] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.650604 7528 ts_tablet_manager.cc:1428] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:57.650820 7529 ts_tablet_manager.cc:1428] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:57.728883 7535 raft_consensus.cc:482] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:57.729080 7535 raft_consensus.cc:504] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.730538 7535 leader_election.cc:288] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:57.735271 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.730977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:34107 (local address 127.28.94.2:43219)
0806 04:29:57.731106 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.731110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.731472 (+ 362us) server_negotiation.cc:407] Connection header received
0806 04:29:57.731674 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.731678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.731730 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.731803 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.732170 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.732882 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.734079 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.734612 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.734720 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.734804 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.735016 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.735104 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:29:57.735584 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.730793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:43219 (local address 127.28.94.3:34107)
0806 04:29:57.731375 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.731390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.731503 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.731924 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.731927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.731945 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.732042 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.732048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.733005 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.733010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.733946 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.733957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.734762 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.734766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.734872 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.734904 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.734984 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.735081 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":284,"thread_start_us":51,"threads_started":1}
I20260806 04:29:57.735622 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.730954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37073 (local address 127.28.94.1:34079)
0806 04:29:57.731090 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.731095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.731381 (+ 286us) server_negotiation.cc:407] Connection header received
0806 04:29:57.731588 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.731592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.731673 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.731741 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:29:57.732006 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.732821 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.734352 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.734927 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.735019 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.735080 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.735394 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.735489 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:29:57.736049 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.730793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34079 (local address 127.28.94.3:37073)
0806 04:29:57.731269 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.731284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.731423 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.731766 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.731769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.731786 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:29:57.731906 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.731911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.732988 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.732993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.734209 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.734223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.735143 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.735147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.735246 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.735271 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.735822 (+ 551us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.735908 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260806 04:29:57.736419 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c06b217a4464adb987de39641e672f1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec" is_pre_election: true
I20260806 04:29:57.736564 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c06b217a4464adb987de39641e672f1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208" is_pre_election: true
I20260806 04:29:57.736744 7060 raft_consensus.cc:2454] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:57.736718 7249 raft_consensus.cc:2454] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:57.737219 7342 leader_election.cc:302] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755; no voters:
I20260806 04:29:57.737695 7535 raft_consensus.cc:2781] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:57.737752 7535 raft_consensus.cc:482] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:57.737789 7535 raft_consensus.cc:3037] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.738793 7535 raft_consensus.cc:504] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.739177 7535 leader_election.cc:288] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Requested vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:57.739563 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c06b217a4464adb987de39641e672f1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208"
I20260806 04:29:57.739621 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c06b217a4464adb987de39641e672f1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec"
I20260806 04:29:57.739720 7060 raft_consensus.cc:3037] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.739727 7249 raft_consensus.cc:3037] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.740921 7249 raft_consensus.cc:2454] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:57.740921 7060 raft_consensus.cc:2454] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:57.741264 7342 leader_election.cc:302] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755; no voters:
I20260806 04:29:57.741456 7535 raft_consensus.cc:2781] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:57.741683 7535 raft_consensus.cc:686] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Becoming Leader. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Running, Role: LEADER
I20260806 04:29:57.741977 7535 consensus_queue.cc:237] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.744925 6860 catalog_manager.cc:5495] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } health_report { overall_health: HEALTHY } } }
Aug 06 04:29:57 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990596, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.62@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:57.788210 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.778828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:32777 (local address 127.0.0.1:56264)
0806 04:29:57.779169 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.779185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.779284 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.779552 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.779554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.779976 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:29:57.780090 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.780097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.781198 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.781200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.781760 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.781768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.782316 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.782319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.782377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.783005 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:29:57.783018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:29:57.784672 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:29:57.786715 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.786727 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.786732 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.786991 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.787252 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:29:57.787254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:29:57.787255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:29:57.787328 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:29:57.787720 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:29:57.787726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:29:57.787838 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.787928 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.788060 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":122,"threads_started":1}
I20260806 04:29:57.788332 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.778925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.28.94.62:32777)
0806 04:29:57.779210 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.779219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.779238 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:29:57.779366 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.779370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.779419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.779515 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:57.780214 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.781100 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.781849 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.782216 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.782305 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.784847 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:57.784851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:57.784854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:57.784881 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:57.786554 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.787083 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.787087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.787089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.787176 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:57.787396 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:57.787399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:57.787401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:57.787633 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:57.787724 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.787997 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.788134 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
I20260806 04:29:57.792475 6860 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:56264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:29:57.793428 6860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:57.798764 7224 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc0369015544ed89efd8c0fbfdb7bc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=392870197f8842889a5a040027ea301d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:57.798785 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc0369015544ed89efd8c0fbfdb7bc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=392870197f8842889a5a040027ea301d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:57.798777 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc0369015544ed89efd8c0fbfdb7bc1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=392870197f8842889a5a040027ea301d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:29:57.799201 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc0369015544ed89efd8c0fbfdb7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.799203 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc0369015544ed89efd8c0fbfdb7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.799301 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc0369015544ed89efd8c0fbfdb7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:57.801895 7529 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: Bootstrap starting.
I20260806 04:29:57.802479 7528 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: Bootstrap starting.
I20260806 04:29:57.802852 7530 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: Bootstrap starting.
I20260806 04:29:57.802889 7529 tablet_bootstrap.cc:654] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.803483 7528 tablet_bootstrap.cc:654] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.803858 7530 tablet_bootstrap.cc:654] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:57.804054 7529 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: No bootstrap required, opened a new log
I20260806 04:29:57.804134 7529 ts_tablet_manager.cc:1397] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:29:57.804589 7529 raft_consensus.cc:348] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.804724 7529 raft_consensus.cc:374] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.804734 7528 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: No bootstrap required, opened a new log
I20260806 04:29:57.804759 7529 raft_consensus.cc:729] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.804834 7528 ts_tablet_manager.cc:1397] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:29:57.804888 7529 consensus_queue.cc:260] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.805258 7530 tablet_bootstrap.cc:492] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: No bootstrap required, opened a new log
I20260806 04:29:57.805197 7548 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.805334 7548 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.805337 7530 ts_tablet_manager.cc:1397] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:57.805317 7528 raft_consensus.cc:348] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.805421 7528 raft_consensus.cc:374] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.805455 7528 raft_consensus.cc:729] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6e46e995a34df08c8e2cd45ab44208, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.805574 7528 consensus_queue.cc:260] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.805814 7529 ts_tablet_manager.cc:1428] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 04:29:57.805785 7530 raft_consensus.cc:348] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.805862 7528 ts_tablet_manager.cc:1428] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:57.805886 7530 raft_consensus.cc:374] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:57.805915 7530 raft_consensus.cc:729] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Initialized, Role: FOLLOWER
I20260806 04:29:57.805972 7534 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.806038 7530 consensus_queue.cc:260] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.806121 7534 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.806267 7535 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.806401 7535 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.806274 7530 ts_tablet_manager.cc:1428] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:57.934880 7535 raft_consensus.cc:482] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:57.935062 7535 raft_consensus.cc:504] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.935524 7535 leader_election.cc:288] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:57.936100 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc0369015544ed89efd8c0fbfdb7bc1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec" is_pre_election: true
I20260806 04:29:57.936108 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc0369015544ed89efd8c0fbfdb7bc1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208" is_pre_election: true
I20260806 04:29:57.936259 7249 raft_consensus.cc:2454] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:57.936262 7060 raft_consensus.cc:2454] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:57.936599 7342 leader_election.cc:302] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755, ecfcef129e604c1fa39f75f801a2ddec; no voters:
I20260806 04:29:57.936795 7535 raft_consensus.cc:2781] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:57.936856 7535 raft_consensus.cc:482] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:57.936882 7535 raft_consensus.cc:3037] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.938081 7535 raft_consensus.cc:504] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.938464 7535 leader_election.cc:288] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Requested vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:57.938913 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc0369015544ed89efd8c0fbfdb7bc1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208"
I20260806 04:29:57.939045 7060 raft_consensus.cc:3037] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.940057 7060 raft_consensus.cc:2454] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:57.940382 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc0369015544ed89efd8c0fbfdb7bc1" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec"
I20260806 04:29:57.940409 7342 leader_election.cc:302] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755; no voters:
I20260806 04:29:57.940524 7249 raft_consensus.cc:3037] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:57.940598 7535 raft_consensus.cc:2781] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:57.940711 7535 raft_consensus.cc:686] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Becoming Leader. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Running, Role: LEADER
I20260806 04:29:57.940871 7535 consensus_queue.cc:237] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:57.941733 7249 raft_consensus.cc:2454] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:57.941722 7571 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: TxnStatusTablet state changed. Reason: New leader 8bebb37b2d384532b5a92a6c67d88755. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.941855 7571 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:57.942230 7593 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:29:57.942414 7593 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:29:57.943029 6857 catalog_manager.cc:5495] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:57.966388 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.962349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.28.94.3:40499)
0806 04:29:57.962467 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.962470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.962484 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:29:57.962655 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.962658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.962718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.962803 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:57.963109 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.963642 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.964847 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.965100 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.965182 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.965547 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.966015 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.966102 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.966203 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.966262 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:29:57.966449 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.962215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40499 (local address 127.0.0.1:60840)
0806 04:29:57.962375 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.962389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.962503 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.962838 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.962841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.962852 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:57.962999 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.963006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.963741 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.963745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.964739 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.964751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.965224 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.965228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.965339 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.965409 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.966103 (+ 694us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.966119 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.966188 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.966296 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 04:29:57.969719 7060 raft_consensus.cc:1261] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:57.969884 7249 raft_consensus.cc:1261] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:57.970345 7535 consensus_queue.cc:1035] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:57.970522 7571 consensus_queue.cc:1035] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:57.972420 7534 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: TxnStatusTablet state changed. Reason: New leader 8bebb37b2d384532b5a92a6c67d88755. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.972553 7534 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.973088 7548 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: TxnStatusTablet state changed. Reason: New leader 8bebb37b2d384532b5a92a6c67d88755. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.973220 7548 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.975796 7535 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.975926 7535 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:57.976017 7599 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.976154 7599 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 8bebb37b2d384532b5a92a6c67d88755: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:29:57.976704 7534 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.976822 7534 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P 2a6e46e995a34df08c8e2cd45ab44208: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.978433 7543 mvcc.cc:205] Tried to move back new op lower bound from 7315417489278132224 to 7315417489167585280. Current Snapshot: MvccSnapshot[applied={T|T < 7315417489278132224}]
I20260806 04:29:57.979424 7548 tablet_replica.cc:439] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } } }
I20260806 04:29:57.979552 7548 tablet_replica.cc:442] T 9fc0369015544ed89efd8c0fbfdb7bc1 P ecfcef129e604c1fa39f75f801a2ddec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:29:57.984248 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.981221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.28.94.3:40499)
0806 04:29:57.981323 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.981327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.981381 (+ 54us) server_negotiation.cc:407] Connection header received
0806 04:29:57.981529 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.981532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.981583 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.981650 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:57.981894 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.982413 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.983233 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.983458 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.983546 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.983808 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.983892 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.983947 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.984047 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.984103 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:29:57.997246 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.993199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37157 (local address 127.28.94.3:40499)
0806 04:29:57.993324 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:57.993328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:57.993637 (+ 309us) server_negotiation.cc:407] Connection header received
0806 04:29:57.993773 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:57.993776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:57.993822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:57.993894 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:57.994178 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.994732 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.996040 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.996269 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.996371 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.996748 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.996835 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.996949 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:57.997035 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 04:29:57.997095 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:29:57.997313 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:57.993106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40499 (local address 127.28.94.3:37157)
0806 04:29:57.993540 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:57.993555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:29:57.993687 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:29:57.993920 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:29:57.993923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:29:57.993931 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:29:57.994085 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.994091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.994833 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.994836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.995873 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:29:57.995886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:29:57.996390 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:29:57.996394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:29:57.996498 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:57.996569 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:29:57.996949 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:29:57.996955 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:29:57.997021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:29:57.997119 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":122,"threads_started":1}
I20260806 04:29:58.004971 7249 raft_consensus.cc:1261] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.005117 7060 raft_consensus.cc:1261] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.005455 7599 consensus_queue.cc:1035] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.005642 7571 consensus_queue.cc:1035] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.053654 6857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.053874 6857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.056367 6857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Sending DeleteTablet for 3 replicas of tablet 1c06b217a4464adb987de39641e672f1
I20260806 04:29:58.057129 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c06b217a4464adb987de39641e672f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260806 04:29:58.057221 7224 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c06b217a4464adb987de39641e672f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35918
I20260806 04:29:58.057354 7557 tablet_replica.cc:330] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: stopping tablet replica
I20260806 04:29:58.057426 7558 tablet_replica.cc:330] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: stopping tablet replica
I20260806 04:29:58.057476 7557 raft_consensus.cc:2229] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.057548 7558 raft_consensus.cc:2229] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.057556 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c06b217a4464adb987de39641e672f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260806 04:29:58.057610 7557 raft_consensus.cc:2258] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.057683 7558 raft_consensus.cc:2258] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.057780 7560 tablet_replica.cc:330] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: stopping tablet replica
I20260806 04:29:58.057904 7560 raft_consensus.cc:2229] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:58.058259 7560 raft_consensus.cc:2258] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.058496 7557 ts_tablet_manager.cc:1905] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.058619 7558 ts_tablet_manager.cc:1905] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.061607 7560 ts_tablet_manager.cc:1905] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.061725 7558 ts_tablet_manager.cc:1918] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:58.061805 7558 log.cc:1196] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/1c06b217a4464adb987de39641e672f1
I20260806 04:29:58.062054 7558 ts_tablet_manager.cc:1939] T 1c06b217a4464adb987de39641e672f1 P ecfcef129e604c1fa39f75f801a2ddec: Deleting consensus metadata
I20260806 04:29:58.062700 7557 ts_tablet_manager.cc:1918] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:58.062772 7557 log.cc:1196] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/1c06b217a4464adb987de39641e672f1
I20260806 04:29:58.063009 7557 ts_tablet_manager.cc:1939] T 1c06b217a4464adb987de39641e672f1 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting consensus metadata
I20260806 04:29:58.063110 6839 catalog_manager.cc:4841] TS ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219): tablet 1c06b217a4464adb987de39641e672f1 (table test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]) successfully deleted
I20260806 04:29:58.063819 6838 catalog_manager.cc:4841] TS 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079): tablet 1c06b217a4464adb987de39641e672f1 (table test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]) successfully deleted
I20260806 04:29:58.064590 7560 ts_tablet_manager.cc:1918] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:58.064679 7560 log.cc:1196] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/1c06b217a4464adb987de39641e672f1
I20260806 04:29:58.064921 7560 ts_tablet_manager.cc:1939] T 1c06b217a4464adb987de39641e672f1 P 8bebb37b2d384532b5a92a6c67d88755: Deleting consensus metadata
I20260806 04:29:58.065665 6841 catalog_manager.cc:4841] TS 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499): tablet 1c06b217a4464adb987de39641e672f1 (table test-table [id=507a7c0ccb9b405c8da0222a7b9f6e18]) successfully deleted
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[6800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:29:58.090901 7585 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:29:58.083252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.28.94.62:32777)
0806 04:29:58.083418 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:58.083423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:58.083571 (+ 148us) server_negotiation.cc:407] Connection header received
0806 04:29:58.083687 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:58.083691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:58.083749 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:58.083854 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:58.084597 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.085305 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.086562 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.086827 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.086911 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:58.090756 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56268: BlockingRecv error: recv got EOF from 127.0.0.1:56268 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[6800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM
I20260806 04:29:58.101706 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:58.092740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.28.94.62:32777)
0806 04:29:58.092918 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:58.092924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:58.093083 (+ 159us) server_negotiation.cc:407] Connection header received
0806 04:29:58.093187 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:58.093191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:58.093250 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:58.093348 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:29:58.093946 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.094733 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.095490 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.095825 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.095920 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:58.098467 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:29:58.098471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:29:58.098474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:29:58.098509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:29:58.100054 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:58.100479 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:58.100483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:58.100485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:58.100582 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:29:58.100792 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:29:58.100795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:29:58.100797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:29:58.101058 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:29:58.101196 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:58.101384 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:29:58.101527 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:29:58.104615 6857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:29:58.105185 6857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:58.110469 7403 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 (DEFAULT_TABLE table=test-table [id=ccb207cb35ab47d790b05ff63371d34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.110469 7224 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 (DEFAULT_TABLE table=test-table [id=ccb207cb35ab47d790b05ff63371d34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.110469 7033 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 (DEFAULT_TABLE table=test-table [id=ccb207cb35ab47d790b05ff63371d34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.110826 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4bffe19824543a5faea8ab3e4b959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.110826 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4bffe19824543a5faea8ab3e4b959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.110827 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4bffe19824543a5faea8ab3e4b959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.113379 7528 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Bootstrap starting.
I20260806 04:29:58.113443 7529 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Bootstrap starting.
I20260806 04:29:58.113462 7530 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Bootstrap starting.
I20260806 04:29:58.114339 7529 tablet_bootstrap.cc:654] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.114339 7530 tablet_bootstrap.cc:654] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.114387 7528 tablet_bootstrap.cc:654] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.116259 7529 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: No bootstrap required, opened a new log
I20260806 04:29:58.116360 7529 ts_tablet_manager.cc:1397] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:58.116935 7530 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: No bootstrap required, opened a new log
I20260806 04:29:58.116809 7529 raft_consensus.cc:348] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.116986 7529 raft_consensus.cc:374] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.117018 7529 raft_consensus.cc:729] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.117030 7530 ts_tablet_manager.cc:1397] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 04:29:58.117161 7529 consensus_queue.cc:260] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.117408 7529 ts_tablet_manager.cc:1428] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:58.117424 7528 tablet_bootstrap.cc:492] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: No bootstrap required, opened a new log
I20260806 04:29:58.117514 7528 ts_tablet_manager.cc:1397] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 04:29:58.117475 7530 raft_consensus.cc:348] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.117584 7530 raft_consensus.cc:374] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.117615 7530 raft_consensus.cc:729] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.117743 7530 consensus_queue.cc:260] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.118000 7530 ts_tablet_manager.cc:1428] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:58.117988 7528 raft_consensus.cc:348] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.118096 7528 raft_consensus.cc:374] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.118125 7528 raft_consensus.cc:729] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6e46e995a34df08c8e2cd45ab44208, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.118244 7528 consensus_queue.cc:260] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.118525 7528 ts_tablet_manager.cc:1428] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:58.443236 7599 raft_consensus.cc:482] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:58.443401 7599 raft_consensus.cc:504] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.443841 7599 leader_election.cc:288] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:58.444376 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4bffe19824543a5faea8ab3e4b959" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec" is_pre_election: true
I20260806 04:29:58.444401 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4bffe19824543a5faea8ab3e4b959" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208" is_pre_election: true
I20260806 04:29:58.444547 7248 raft_consensus.cc:2454] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:58.444564 7060 raft_consensus.cc:2454] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:58.444909 7342 leader_election.cc:302] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755; no voters:
I20260806 04:29:58.445118 7599 raft_consensus.cc:2781] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:58.445182 7599 raft_consensus.cc:482] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:58.445209 7599 raft_consensus.cc:3037] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.446321 7599 raft_consensus.cc:504] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.446713 7599 leader_election.cc:288] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Requested vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:58.447140 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4bffe19824543a5faea8ab3e4b959" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec"
I20260806 04:29:58.447184 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4bffe19824543a5faea8ab3e4b959" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208"
I20260806 04:29:58.447296 7248 raft_consensus.cc:3037] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.447302 7060 raft_consensus.cc:3037] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.448493 7248 raft_consensus.cc:2454] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:58.448493 7060 raft_consensus.cc:2454] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:58.448829 7343 leader_election.cc:302] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bebb37b2d384532b5a92a6c67d88755, ecfcef129e604c1fa39f75f801a2ddec; no voters:
I20260806 04:29:58.449105 7599 raft_consensus.cc:2781] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:58.449191 7599 raft_consensus.cc:686] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Becoming Leader. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Running, Role: LEADER
I20260806 04:29:58.449347 7599 consensus_queue.cc:237] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.452023 6860 catalog_manager.cc:5495] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:58.489435 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:58.486614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.28.94.3:40499)
0806 04:29:58.486751 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:29:58.486755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:29:58.486771 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:29:58.486819 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:29:58.486821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:29:58.486867 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:29:58.486929 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:29:58.487151 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.487656 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.488484 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:29:58.488705 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:29:58.488785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:29:58.489038 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:29:58.489119 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:29:58.489166 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 04:29:58.489232 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 04:29:58.489280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:29:58.491696 7060 raft_consensus.cc:1261] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.491691 7248 raft_consensus.cc:1261] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.492156 7599 consensus_queue.cc:1035] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.492350 7535 consensus_queue.cc:1035] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.500389 6860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.500568 6860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.502549 6860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Sending DeleteTablet for 3 replicas of tablet e5a4bffe19824543a5faea8ab3e4b959
I20260806 04:29:58.503176 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260806 04:29:58.503185 7224 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35918
I20260806 04:29:58.503387 7557 tablet_replica.cc:330] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: stopping tablet replica
I20260806 04:29:58.503520 7557 raft_consensus.cc:2229] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.503505 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4bffe19824543a5faea8ab3e4b959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260806 04:29:58.503687 7560 tablet_replica.cc:330] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: stopping tablet replica
I20260806 04:29:58.503721 7557 raft_consensus.cc:2258] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.503806 7560 raft_consensus.cc:2229] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:58.503926 7558 tablet_replica.cc:330] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: stopping tablet replica
I20260806 04:29:58.503844 6860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:29:58.504042 7560 raft_consensus.cc:2258] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.504051 7558 raft_consensus.cc:2229] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.504179 7558 raft_consensus.cc:2258] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 04:29:58.504314 6860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:29:58.504863 7558 ts_tablet_manager.cc:1905] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.504863 7560 ts_tablet_manager.cc:1905] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.504997 7557 ts_tablet_manager.cc:1905] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.507454 7557 ts_tablet_manager.cc:1918] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:58.507459 7560 ts_tablet_manager.cc:1918] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:58.507450 7558 ts_tablet_manager.cc:1918] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:29:58.507521 7557 log.cc:1196] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/e5a4bffe19824543a5faea8ab3e4b959
I20260806 04:29:58.507534 7560 log.cc:1196] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/e5a4bffe19824543a5faea8ab3e4b959
I20260806 04:29:58.507534 7558 log.cc:1196] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/e5a4bffe19824543a5faea8ab3e4b959
I20260806 04:29:58.507718 7557 ts_tablet_manager.cc:1939] T e5a4bffe19824543a5faea8ab3e4b959 P 2a6e46e995a34df08c8e2cd45ab44208: Deleting consensus metadata
I20260806 04:29:58.507786 7558 ts_tablet_manager.cc:1939] T e5a4bffe19824543a5faea8ab3e4b959 P ecfcef129e604c1fa39f75f801a2ddec: Deleting consensus metadata
I20260806 04:29:58.507786 7560 ts_tablet_manager.cc:1939] T e5a4bffe19824543a5faea8ab3e4b959 P 8bebb37b2d384532b5a92a6c67d88755: Deleting consensus metadata
I20260806 04:29:58.508683 6838 catalog_manager.cc:4841] TS 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079): tablet e5a4bffe19824543a5faea8ab3e4b959 (table test-table [id=ccb207cb35ab47d790b05ff63371d34a]) successfully deleted
I20260806 04:29:58.509286 6839 catalog_manager.cc:4841] TS ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219): tablet e5a4bffe19824543a5faea8ab3e4b959 (table test-table [id=ccb207cb35ab47d790b05ff63371d34a]) successfully deleted
I20260806 04:29:58.509546 6841 catalog_manager.cc:4841] TS 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499): tablet e5a4bffe19824543a5faea8ab3e4b959 (table test-table [id=ccb207cb35ab47d790b05ff63371d34a]) successfully deleted
I20260806 04:29:58.510866 7033 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 (DEFAULT_TABLE table=test-table [id=80beac08c2854501b5c0d402a0759cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.510955 7224 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 (DEFAULT_TABLE table=test-table [id=80beac08c2854501b5c0d402a0759cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.511163 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3e047eebc5446ca78a2c3a5b31ec41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.511232 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3e047eebc5446ca78a2c3a5b31ec41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.513229 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 (DEFAULT_TABLE table=test-table [id=80beac08c2854501b5c0d402a0759cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:29:58.513402 7528 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: Bootstrap starting.
I20260806 04:29:58.513535 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3e047eebc5446ca78a2c3a5b31ec41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:29:58.513629 7529 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Bootstrap starting.
I20260806 04:29:58.514469 7529 tablet_bootstrap.cc:654] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.514634 7528 tablet_bootstrap.cc:654] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.515869 7528 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: No bootstrap required, opened a new log
I20260806 04:29:58.515935 7528 ts_tablet_manager.cc:1397] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:29:58.516041 7529 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: No bootstrap required, opened a new log
I20260806 04:29:58.516093 7530 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Bootstrap starting.
I20260806 04:29:58.516139 7529 ts_tablet_manager.cc:1397] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 04:29:58.516419 7528 raft_consensus.cc:348] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.516567 7528 raft_consensus.cc:374] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.516601 7528 raft_consensus.cc:729] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6e46e995a34df08c8e2cd45ab44208, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.516533 7529 raft_consensus.cc:348] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.516646 7529 raft_consensus.cc:374] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.516675 7529 raft_consensus.cc:729] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfcef129e604c1fa39f75f801a2ddec, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.516736 7528 consensus_queue.cc:260] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.516795 7529 consensus_queue.cc:260] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.517056 7528 ts_tablet_manager.cc:1428] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:29:58.517094 7529 ts_tablet_manager.cc:1428] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:29:58.517148 7530 tablet_bootstrap.cc:654] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Neither blocks nor log segments found. Creating new log.
I20260806 04:29:58.518362 7530 tablet_bootstrap.cc:492] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: No bootstrap required, opened a new log
I20260806 04:29:58.518457 7530 ts_tablet_manager.cc:1397] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 04:29:58.518880 7530 raft_consensus.cc:348] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.519022 7530 raft_consensus.cc:374] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:29:58.519062 7530 raft_consensus.cc:729] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Initialized, Role: FOLLOWER
I20260806 04:29:58.519187 7530 consensus_queue.cc:260] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.519449 7530 ts_tablet_manager.cc:1428] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:29:58.777433 7599 raft_consensus.cc:482] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:29:58.777650 7599 raft_consensus.cc:504] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.778142 7599 leader_election.cc:288] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:58.778724 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3e047eebc5446ca78a2c3a5b31ec41" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208" is_pre_election: true
I20260806 04:29:58.778728 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3e047eebc5446ca78a2c3a5b31ec41" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec" is_pre_election: true
I20260806 04:29:58.778898 7060 raft_consensus.cc:2454] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:58.778918 7248 raft_consensus.cc:2454] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 0.
I20260806 04:29:58.779284 7343 leader_election.cc:302] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bebb37b2d384532b5a92a6c67d88755, ecfcef129e604c1fa39f75f801a2ddec; no voters:
I20260806 04:29:58.779488 7599 raft_consensus.cc:2781] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:29:58.779560 7599 raft_consensus.cc:482] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:29:58.779589 7599 raft_consensus.cc:3037] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.780611 7599 raft_consensus.cc:504] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.781036 7599 leader_election.cc:288] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Requested vote from peers 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079), ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219)
I20260806 04:29:58.781417 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3e047eebc5446ca78a2c3a5b31ec41" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a6e46e995a34df08c8e2cd45ab44208"
I20260806 04:29:58.781418 7248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3e047eebc5446ca78a2c3a5b31ec41" candidate_uuid: "8bebb37b2d384532b5a92a6c67d88755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfcef129e604c1fa39f75f801a2ddec"
I20260806 04:29:58.781549 7248 raft_consensus.cc:3037] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.781548 7060 raft_consensus.cc:3037] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:29:58.782523 7060 raft_consensus.cc:2454] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:58.782533 7248 raft_consensus.cc:2454] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bebb37b2d384532b5a92a6c67d88755 in term 1.
I20260806 04:29:58.782831 7342 leader_election.cc:302] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6e46e995a34df08c8e2cd45ab44208, 8bebb37b2d384532b5a92a6c67d88755; no voters:
I20260806 04:29:58.783013 7599 raft_consensus.cc:2781] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:29:58.783113 7599 raft_consensus.cc:686] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Becoming Leader. State: Replica: 8bebb37b2d384532b5a92a6c67d88755, State: Running, Role: LEADER
I20260806 04:29:58.783258 7599 consensus_queue.cc:237] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } }
I20260806 04:29:58.785055 6861 catalog_manager.cc:5495] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "8bebb37b2d384532b5a92a6c67d88755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bebb37b2d384532b5a92a6c67d88755" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40499 } health_report { overall_health: HEALTHY } } }
I20260806 04:29:58.825431 7248 raft_consensus.cc:1261] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.825519 7060 raft_consensus.cc:1261] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Refusing update from remote peer 8bebb37b2d384532b5a92a6c67d88755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:29:58.825904 7599 consensus_queue.cc:1035] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecfcef129e604c1fa39f75f801a2ddec" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.826087 7535 consensus_queue.cc:1035] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a6e46e995a34df08c8e2cd45ab44208" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:29:58.859355 6860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.859522 6860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:29:58.861886 6860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ac4e4788d6946ccbedbd4ea387fed9f: Sending DeleteTablet for 3 replicas of tablet 1e3e047eebc5446ca78a2c3a5b31ec41
I20260806 04:29:58.862589 7033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46358
I20260806 04:29:58.862653 7224 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35918
I20260806 04:29:58.862823 7557 tablet_replica.cc:330] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208: stopping tablet replica
I20260806 04:29:58.862845 7558 tablet_replica.cc:330] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: stopping tablet replica
I20260806 04:29:58.862957 7557 raft_consensus.cc:2229] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.862969 7558 raft_consensus.cc:2229] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:29:58.863095 7558 raft_consensus.cc:2258] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.863101 7557 raft_consensus.cc:2258] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 2a6e46e995a34df08c8e2cd45ab44208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.863304 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 6952
I20260806 04:29:58.863451 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3e047eebc5446ca78a2c3a5b31ec41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:29:58 UTC) from {username='oryx', principal='oryx/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260806 04:29:58.864447 7558 ts_tablet_manager.cc:1905] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.866755 7558 ts_tablet_manager.cc:1918] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:58.866832 7558 log.cc:1196] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/1e3e047eebc5446ca78a2c3a5b31ec41
I20260806 04:29:58.867033 7558 ts_tablet_manager.cc:1939] T 1e3e047eebc5446ca78a2c3a5b31ec41 P ecfcef129e604c1fa39f75f801a2ddec: Deleting consensus metadata
I20260806 04:29:58.867847 6839 catalog_manager.cc:4841] TS ecfcef129e604c1fa39f75f801a2ddec (127.28.94.2:43219): tablet 1e3e047eebc5446ca78a2c3a5b31ec41 (table test-table [id=80beac08c2854501b5c0d402a0759cb0]) successfully deleted
I20260806 04:29:58.871104 7560 tablet_replica.cc:330] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: stopping tablet replica
I20260806 04:29:58.871290 7560 raft_consensus.cc:2229] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:29:58.871539 7560 raft_consensus.cc:2258] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:29:58.873595 7560 ts_tablet_manager.cc:1905] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:29:58.875850 7560 ts_tablet_manager.cc:1918] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:29:58.875936 7560 log.cc:1196] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/1e3e047eebc5446ca78a2c3a5b31ec41
I20260806 04:29:58.876150 7560 ts_tablet_manager.cc:1939] T 1e3e047eebc5446ca78a2c3a5b31ec41 P 8bebb37b2d384532b5a92a6c67d88755: Deleting consensus metadata
I20260806 04:29:58.877048 6841 catalog_manager.cc:4841] TS 8bebb37b2d384532b5a92a6c67d88755 (127.28.94.3:40499): tablet 1e3e047eebc5446ca78a2c3a5b31ec41 (table test-table [id=80beac08c2854501b5c0d402a0759cb0]) successfully deleted
W20260806 04:29:58.880730 6838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:34079 (error 108)
I20260806 04:29:58.880970 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 7141
I20260806 04:29:58.886252 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:29:58.881635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:34079 (local address 127.0.0.1:49726)
0806 04:29:58.885986 (+ 4351us) negotiation.cc:107] Waiting for socket to connect
0806 04:29:58.886115 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4229,"thread_start_us":1378,"threads_started":1}
W20260806 04:29:58.886420 6838 catalog_manager.cc:4584] TS 2a6e46e995a34df08c8e2cd45ab44208 (127.28.94.1:34079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e3e047eebc5446ca78a2c3a5b31ec41: Network error: Client connection negotiation failed: client connection to 127.28.94.1:34079: connect: Connection refused (error 111)
I20260806 04:29:58.896551 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 7322
I20260806 04:29:58.910386 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 6819
2026-08-06T04:29:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4022 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[7652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[7652](info): set up 2 sockets
Aug 06 04:29:58 dist-test-slave-ncp9 krb5kdc[7652](info): commencing operation
krb5kdc: starting...
W20260806 04:30:00.034178 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:30:00 dist-test-slave-ncp9 krb5kdc[7652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:30:00.173437 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:42693
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:00.300644 7671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:00.301004 7671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:00.301069 7671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:00.305411 7671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:00.305505 7671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:00.305531 7671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:00.305554 7671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:00.305583 7671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:00.311854 7671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:42693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:42693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:00.313228 7671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:00.314570 7671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:00.323964 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:00.324028 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:00.324231 7671 server_base.cc:1034] running on GCE node
W20260806 04:30:00.324254 7677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:00.325062 7671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:00.326345 7671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:00.327528 7671 hybrid_clock.cc:648] HybridClock initialized: now 1785990600327497 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T04:30:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1424 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:30:00 dist-test-slave-ncp9 krb5kdc[7690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:30:00 dist-test-slave-ncp9 krb5kdc[7690](info): set up 2 sockets
Aug 06 04:30:00 dist-test-slave-ncp9 krb5kdc[7690](info): commencing operation
krb5kdc: starting...
W20260806 04:30:01.458797 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:30:01 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:30:01.599473 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40993
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:40993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:01.721462 7707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:01.721791 7707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:01.721833 7707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:01.726012 7707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:01.726090 7707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:01.726115 7707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:01.726132 7707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:01.726146 7707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:01.731817 7707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:40993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:01.733050 7707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:01.734124 7707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:01.743667 7713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:01.743589 7712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:01.743587 7715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:01.743631 7707 server_base.cc:1034] running on GCE node
I20260806 04:30:01.744617 7707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:01.745900 7707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:01.747072 7707 hybrid_clock.cc:648] HybridClock initialized: now 1785990601747060 us; error 36 us; skew 500 ppm
Aug 06 04:30:01 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:01.757151 7707 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:30:01.758358 7707 webserver.cc:460] Webserver started at http://127.28.94.62:44289/ using document root <none> and password file <none>
I20260806 04:30:01.759009 7707 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:01.759091 7707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:01.759334 7707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:01.761780 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "0331ac55f28d40159ed76c29e284bfce"
format_stamp: "Formatted at 2026-08-06 04:30:01 on dist-test-slave-ncp9"
server_key: "4220afaf3d17de363df13fa8867fc4f3"
server_key_iv: "4fb39b961538316fbf96e59c3b646813"
server_key_version: "encryptionkey@0"
I20260806 04:30:01.762447 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "0331ac55f28d40159ed76c29e284bfce"
format_stamp: "Formatted at 2026-08-06 04:30:01 on dist-test-slave-ncp9"
server_key: "4220afaf3d17de363df13fa8867fc4f3"
server_key_iv: "4fb39b961538316fbf96e59c3b646813"
server_key_version: "encryptionkey@0"
I20260806 04:30:01.767113 7707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:01.770262 7723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:01.771847 7707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:30:01.772040 7707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "0331ac55f28d40159ed76c29e284bfce"
format_stamp: "Formatted at 2026-08-06 04:30:01 on dist-test-slave-ncp9"
server_key: "4220afaf3d17de363df13fa8867fc4f3"
server_key_iv: "4fb39b961538316fbf96e59c3b646813"
server_key_version: "encryptionkey@0"
I20260806 04:30:01.772179 7707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:01.785682 7707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:01.789721 7707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:01.789937 7707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:01.798116 7707 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:40993
I20260806 04:30:01.798483 7793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:40993 every 8 connection(s)
I20260806 04:30:01.799295 7707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:30:01.802600 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:01.807946 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 7707
I20260806 04:30:01.808096 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:30:01.808383 29048 external_mini_cluster.cc:1428] Setting key 680a8585173df41c17db1582ac55eed9
I20260806 04:30:01.809866 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Bootstrap starting.
I20260806 04:30:01.812734 7794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:01.813696 7794 log.cc:824] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:01.816308 7794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: No bootstrap required, opened a new log
Aug 06 04:30:01 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:01.821111 7794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:01.821497 7794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:01.821601 7794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0331ac55f28d40159ed76c29e284bfce, State: Initialized, Role: FOLLOWER
I20260806 04:30:01.822170 7794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:01.822345 7794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:30:01.822418 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:30:01.822522 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:01.823568 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:01.824123 7794 leader_election.cc:302] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0331ac55f28d40159ed76c29e284bfce; no voters:
I20260806 04:30:01.824535 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:30:01.824636 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:30:01.824882 7799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 LEADER]: Becoming Leader. State: Replica: 0331ac55f28d40159ed76c29e284bfce, State: Running, Role: LEADER
I20260806 04:30:01.825341 7799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:01.825840 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:01.809997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.28.94.62:40993)
0806 04:30:01.810533 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:01.810578 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:01.810627 (+ 49us) server_negotiation.cc:407] Connection header received
0806 04:30:01.811361 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:01.811380 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:01.811755 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:01.812066 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:01.813366 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:01.814513 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:01.815790 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:01.816315 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:01.816505 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:01.819964 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:01.819972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:01.819991 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:01.820033 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:01.822254 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:01.822846 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:01.822853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:01.822858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:01.822993 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:01.823281 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:01.823284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:01.823285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:01.823762 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:01.823959 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:01.824287 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 04:30:01.824668 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":129,"threads_started":1}
I20260806 04:30:01.825975 7794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:30:01.827616 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0331ac55f28d40159ed76c29e284bfce. Latest consensus state: current_term: 1 leader_uuid: "0331ac55f28d40159ed76c29e284bfce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } } }
I20260806 04:30:01.827757 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:01.829097 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0331ac55f28d40159ed76c29e284bfce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } } }
I20260806 04:30:01.829228 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: This master's current role is: LEADER
W20260806 04:30:01.832553 7822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:30:01.832654 7822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:30:01.832734 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:30:01.833638 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:30:01.840317 7816 catalog_manager.cc:1347] Generated new cluster ID: 8cb27a3dffea4bffaf6f1d14ff140ff8
I20260806 04:30:01.840422 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:30:01.874202 7816 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:30:01.875283 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:30:01.884334 7816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Generated new TSK 0
I20260806 04:30:01.885265 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:30:01.928557 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:02.051471 7828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:02.051802 7828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:02.051849 7828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:02.056126 7828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:02.056202 7828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:02.056267 7828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:30:02.061990 7828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:02.063195 7828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:02.064260 7828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:02.074884 7839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.074887 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.075222 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:02.075238 7828 server_base.cc:1034] running on GCE node
I20260806 04:30:02.075794 7828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:02.076551 7828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:02.077754 7828 hybrid_clock.cc:648] HybridClock initialized: now 1785990602077696 us; error 68 us; skew 500 ppm
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:02.080936 7828 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:30:02.082171 7828 webserver.cc:460] Webserver started at http://127.28.94.1:43297/ using document root <none> and password file <none>
I20260806 04:30:02.082834 7828 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:02.082909 7828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:02.083137 7828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:02.085611 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "12059a9bc2304edfb052e50928569f8f"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "01791acd4329a6733cfb0cfcde212785"
server_key_iv: "f9cf45c5113b0fce840b3bcf5fb60f27"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.086280 7828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "12059a9bc2304edfb052e50928569f8f"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "01791acd4329a6733cfb0cfcde212785"
server_key_iv: "f9cf45c5113b0fce840b3bcf5fb60f27"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.090653 7828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:02.093734 7848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.095234 7828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:02.095379 7828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "12059a9bc2304edfb052e50928569f8f"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "01791acd4329a6733cfb0cfcde212785"
server_key_iv: "f9cf45c5113b0fce840b3bcf5fb60f27"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.095502 7828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:02.107697 7828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:02.111657 7828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:02.112004 7828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:02.112972 7828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:02.114022 7828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:02.114080 7828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.114141 7828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:02.114171 7828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.123906 7828 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:34271
I20260806 04:30:02.123924 7997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:34271 every 8 connection(s)
I20260806 04:30:02.124941 7828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:30:02.127000 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 7828
I20260806 04:30:02.127147 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:30:02.127461 29048 external_mini_cluster.cc:1428] Setting key 2b5330e769038c5916d126d6f40b0daf
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:30:02.139920 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.126955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:48345 (local address 127.28.94.62:40993)
0806 04:30:02.127113 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:02.127118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:02.128253 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 04:30:02.129747 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:02.129751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:02.129810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:02.129881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:02.131160 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.131770 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.133000 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.133253 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.133345 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.136120 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:02.136124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:02.136126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:02.136155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:02.137643 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.138241 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.138243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.138244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.138324 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.138679 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.138684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.138686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.138936 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:02.139054 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:02.139622 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 04:30:02.139764 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:30:02.140883 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.127515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:48345)
0806 04:30:02.128106 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:02.128165 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:02.129508 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:02.130067 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:02.130075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:02.130449 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:02.130994 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.131009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.131913 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.131916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.132835 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.132847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.133422 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.133426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.133604 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.134304 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:02.134322 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:02.135901 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:02.137784 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.137794 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.137818 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.138155 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.138392 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.138394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.138396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.138545 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.139071 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:02.139078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:02.139368 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 04:30:02.139572 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 04:30:02.139899 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:30:02.142458 7998 heartbeater.cc:347] Connected to a master server at 127.28.94.62:40993
I20260806 04:30:02.142735 7998 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:02.143297 7998 heartbeater.cc:511] Master 127.28.94.62:40993 requested a full tablet report, sending...
I20260806 04:30:02.145076 7744 ts_manager.cc:194] Registered new tserver with Master: 12059a9bc2304edfb052e50928569f8f (127.28.94.1:34271)
I20260806 04:30:02.146296 7744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:48345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:30:02.160157 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:02.284976 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:02.285341 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:02.285398 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:02.290073 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:02.290163 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:02.290227 8004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:30:02.296375 8004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:02.297734 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:02.298938 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:02.309649 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.309643 8016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.309643 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:02.309705 8004 server_base.cc:1034] running on GCE node
I20260806 04:30:02.310406 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:02.311164 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:02.312335 8004 hybrid_clock.cc:648] HybridClock initialized: now 1785990602312311 us; error 48 us; skew 500 ppm
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:02.315300 8004 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:30:02.316367 8004 webserver.cc:460] Webserver started at http://127.28.94.2:39101/ using document root <none> and password file <none>
I20260806 04:30:02.317044 8004 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:02.317124 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:02.317387 8004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:02.319757 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "bfe3e15bbbb64c729d7f50b89906db5d"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "bc58644410269e00ec1b7f3d9df6c30b"
server_key_iv: "d60e520e54a5643deaf289a4ccf1fb0a"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.320369 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "bfe3e15bbbb64c729d7f50b89906db5d"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "bc58644410269e00ec1b7f3d9df6c30b"
server_key_iv: "d60e520e54a5643deaf289a4ccf1fb0a"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.324595 8004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260806 04:30:02.327509 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.329073 8004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:02.329226 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "bfe3e15bbbb64c729d7f50b89906db5d"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "bc58644410269e00ec1b7f3d9df6c30b"
server_key_iv: "d60e520e54a5643deaf289a4ccf1fb0a"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.329347 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:02.344225 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:02.347476 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:02.347661 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:02.348287 8004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:02.349279 8004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:02.349334 8004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.349392 8004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:02.349418 8004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.358855 8004 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43443
I20260806 04:30:02.358894 8181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43443 every 8 connection(s)
I20260806 04:30:02.359925 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:30:02.368952 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8004
I20260806 04:30:02.369117 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:30:02.369401 29048 external_mini_cluster.cc:1428] Setting key 96724e6e3a0cb42ac6315517b7dce921
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:02.374883 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.362016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:55461 (local address 127.28.94.62:40993)
0806 04:30:02.362150 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:02.362154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:02.363121 (+ 967us) server_negotiation.cc:407] Connection header received
0806 04:30:02.364480 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:02.364483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:02.364546 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:02.364624 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:02.366063 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.366641 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.367835 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.368068 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.368153 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.371199 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:02.371202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:02.371205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:02.371232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:02.372582 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.373216 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.373219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.373220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.373304 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.373663 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.373666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.373667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.373872 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:02.373960 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:02.374560 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 04:30:02.374708 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:30:02.375921 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.362413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:55461)
0806 04:30:02.362923 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:02.362972 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:02.364246 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:02.364810 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:02.364817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:02.365271 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:02.365890 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.365907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.366754 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.366756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.367708 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.367718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.368270 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.368274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.368463 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.369287 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:02.369306 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:02.371008 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:02.372723 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.372731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.372752 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.373113 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.373395 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.373398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.373400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.373560 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.373979 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:02.373986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:02.374288 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 04:30:02.374471 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 04:30:02.374868 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260806 04:30:02.377231 8184 heartbeater.cc:347] Connected to a master server at 127.28.94.62:40993
I20260806 04:30:02.377496 8184 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:02.378106 8184 heartbeater.cc:511] Master 127.28.94.62:40993 requested a full tablet report, sending...
I20260806 04:30:02.379382 7744 ts_manager.cc:194] Registered new tserver with Master: bfe3e15bbbb64c729d7f50b89906db5d (127.28.94.2:43443)
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:30:02.380043 7744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:55461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:30:02.400817 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:02.524361 8191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:02.524662 8191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:02.524709 8191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:02.529109 8191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:02.529181 8191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:02.529246 8191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:30:02.534941 8191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:02.536156 8191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:02.537321 8191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:30:02.547871 8191 server_base.cc:1034] running on GCE node
W20260806 04:30:02.547832 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.547824 8199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.547824 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:02.548678 8191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:02.549450 8191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:02.550647 8191 hybrid_clock.cc:648] HybridClock initialized: now 1785990602550613 us; error 47 us; skew 500 ppm
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:02.553719 8191 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:30:02.554765 8191 webserver.cc:460] Webserver started at http://127.28.94.3:34313/ using document root <none> and password file <none>
I20260806 04:30:02.555380 8191 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:02.555456 8191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:02.555688 8191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:02.557991 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "d3072c01e45448aebb3fae612c7e723a"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "6adc3baa350167a2559ebbc34a904d43"
server_key_iv: "288793451f3b6ce627e8c8ebfa5b85d6"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.558631 8191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "d3072c01e45448aebb3fae612c7e723a"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "6adc3baa350167a2559ebbc34a904d43"
server_key_iv: "288793451f3b6ce627e8c8ebfa5b85d6"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.562985 8191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:02.565796 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.567248 8191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:02.567399 8191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "d3072c01e45448aebb3fae612c7e723a"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "6adc3baa350167a2559ebbc34a904d43"
server_key_iv: "288793451f3b6ce627e8c8ebfa5b85d6"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.567528 8191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:02.588146 8191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:02.591496 8191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:02.591701 8191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:02.592386 8191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:02.593417 8191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:02.593470 8191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.593513 8191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:02.593528 8191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.603130 8191 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:34999
I20260806 04:30:02.603190 8358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:34999 every 8 connection(s)
I20260806 04:30:02.604146 8191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:30:02.609545 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8191
I20260806 04:30:02.609686 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:30:02.609977 29048 external_mini_cluster.cc:1428] Setting key 40f611801f2b4d887fb491e960ba6769
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:02.618672 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.606185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:47689 (local address 127.28.94.62:40993)
0806 04:30:02.606346 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:02.606350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:02.607302 (+ 952us) server_negotiation.cc:407] Connection header received
0806 04:30:02.608685 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:02.608689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:02.608744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:02.608821 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:02.610322 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.610932 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.612245 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.612478 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.612570 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.615207 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:02.615211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:02.615212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:02.615242 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:02.616406 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.616986 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.616989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.616990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.617076 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:02.617444 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:02.617446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:02.617447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:02.617660 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:02.617748 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:02.618327 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 04:30:02.618485 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:30:02.619583 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:02.606617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:47689)
0806 04:30:02.607134 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:02.607181 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:02.608439 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:02.609109 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:02.609118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:02.609521 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:02.610151 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.610169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.611072 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.611075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.612116 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:02.612129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:02.612641 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:02.612645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:02.612830 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:02.613470 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:02.613500 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:02.615016 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:02.616534 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.616542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.616561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.616841 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.617162 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:02.617166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:02.617168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:02.617341 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:02.617771 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:02.617779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:02.618067 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 04:30:02.618249 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 04:30:02.618657 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260806 04:30:02.620870 8359 heartbeater.cc:347] Connected to a master server at 127.28.94.62:40993
I20260806 04:30:02.621176 8359 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:02.621748 8359 heartbeater.cc:511] Master 127.28.94.62:40993 requested a full tablet report, sending...
I20260806 04:30:02.622948 7744 ts_manager.cc:194] Registered new tserver with Master: d3072c01e45448aebb3fae612c7e723a (127.28.94.3:34999)
I20260806 04:30:02.623715 7744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:47689
I20260806 04:30:02.624212 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
Entry for principal oryx/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab.
WARNING: no policy specified for oryx/127.28.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
Entry for principal oryx/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab.
WARNING: no policy specified for oryx/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
Entry for principal oryx/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab.
I20260806 04:30:02.715873 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 7828
I20260806 04:30:02.731469 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8004
I20260806 04:30:02.742463 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8191
I20260806 04:30:02.760455 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 7707
I20260806 04:30:02.774768 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40993
--webserver_interface=127.28.94.62
--webserver_port=44289
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:40993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:02.896807 8375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:02.897176 8375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:02.897217 8375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:02.901480 8375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:02.901576 8375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:02.901595 8375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:02.901611 8375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:02.901626 8375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:02.907413 8375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:40993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:02.908649 8375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:02.909951 8375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:02.919449 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:02.919443 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:02.919502 8375 server_base.cc:1034] running on GCE node
W20260806 04:30:02.919443 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:02.920369 8375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:02.921677 8375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:02.922883 8375 hybrid_clock.cc:648] HybridClock initialized: now 1785990602922858 us; error 52 us; skew 500 ppm
Aug 06 04:30:02 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990602, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:02.926061 8375 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:30:02.927094 8375 webserver.cc:460] Webserver started at http://127.28.94.62:44289/ using document root <none> and password file <none>
I20260806 04:30:02.927681 8375 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:02.927759 8375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:02.932181 8375 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:30:02.934628 8393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:02.935953 8375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:30:02.936105 8375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "0331ac55f28d40159ed76c29e284bfce"
format_stamp: "Formatted at 2026-08-06 04:30:01 on dist-test-slave-ncp9"
server_key: "4220afaf3d17de363df13fa8867fc4f3"
server_key_iv: "4fb39b961538316fbf96e59c3b646813"
server_key_version: "encryptionkey@0"
I20260806 04:30:02.936673 8375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:02.949826 8375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:02.953347 8375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:02.953529 8375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:02.960942 8375 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:40993
I20260806 04:30:02.961063 8460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:40993 every 8 connection(s)
I20260806 04:30:02.962186 8375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:30:02.962745 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8375
I20260806 04:30:02.963222 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:34271
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:40993
--webserver_port=43297
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:30:02.967424 8461 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:30:02.969476 8461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Bootstrap starting.
I20260806 04:30:02.988538 8461 log.cc:824] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:02.996069 8461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:30:02.996482 8461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Bootstrap complete.
I20260806 04:30:03.001122 8461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:03.001623 8461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0331ac55f28d40159ed76c29e284bfce, State: Initialized, Role: FOLLOWER
I20260806 04:30:03.002156 8461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:03.002319 8461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:30:03.002393 8461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:30:03.002491 8461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:30:03.004729 8461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:03.005141 8461 leader_election.cc:302] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0331ac55f28d40159ed76c29e284bfce; no voters:
I20260806 04:30:03.005489 8461 leader_election.cc:288] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:30:03.005622 8469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:30:03.005818 8469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [term 2 LEADER]: Becoming Leader. State: Replica: 0331ac55f28d40159ed76c29e284bfce, State: Running, Role: LEADER
I20260806 04:30:03.006214 8469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } }
I20260806 04:30:03.006909 8461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:30:03.007824 8471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0331ac55f28d40159ed76c29e284bfce. Latest consensus state: current_term: 2 leader_uuid: "0331ac55f28d40159ed76c29e284bfce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } } }
I20260806 04:30:03.007961 8471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:03.008231 8470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0331ac55f28d40159ed76c29e284bfce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0331ac55f28d40159ed76c29e284bfce" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40993 } } }
I20260806 04:30:03.008306 8470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:03.008332 8480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:30:03.012133 8480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:30:03.013154 8480 catalog_manager.cc:1259] Loaded cluster ID: 8cb27a3dffea4bffaf6f1d14ff140ff8
I20260806 04:30:03.013216 8480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:30:03.015036 8480 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:30:03.015682 8480 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0331ac55f28d40159ed76c29e284bfce: Loaded TSK: 0
I20260806 04:30:03.016739 8480 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:30:03.101084 8467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:03.101392 8467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:03.101438 8467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:03.105762 8467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:03.105846 8467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:03.105914 8467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:30:03.111819 8467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.1.keytab
--principal=oryx/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:34271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=43297
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:03.113080 8467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:03.114241 8467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:03.124962 8503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:03.124960 8502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:03.125213 8467 server_base.cc:1034] running on GCE node
W20260806 04:30:03.125368 8506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:03.125824 8467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:03.126559 8467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:03.127753 8467 hybrid_clock.cc:648] HybridClock initialized: now 1785990603127736 us; error 30 us; skew 500 ppm
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990603, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:03.130897 8467 init.cc:377] Logged in from keytab as oryx/127.28.94.1@KRBTEST.COM (short username oryx)
I20260806 04:30:03.132031 8467 webserver.cc:460] Webserver started at http://127.28.94.1:43297/ using document root <none> and password file <none>
I20260806 04:30:03.132669 8467 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:03.132764 8467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:03.137156 8467 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 04:30:03.139474 8517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.140880 8467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:30:03.141103 8467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "12059a9bc2304edfb052e50928569f8f"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "01791acd4329a6733cfb0cfcde212785"
server_key_iv: "f9cf45c5113b0fce840b3bcf5fb60f27"
server_key_version: "encryptionkey@0"
I20260806 04:30:03.141721 8467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:03.155555 8467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:03.159040 8467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:03.159250 8467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:03.159930 8467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:03.160883 8467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:03.160985 8467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.161046 8467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:03.161069 8467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.170313 8467 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:34271
I20260806 04:30:03.170337 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:34271 every 8 connection(s)
I20260806 04:30:03.171518 8467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:30:03.171834 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8467
I20260806 04:30:03.172376 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:43443
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:40993
--webserver_port=39101
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:30:03.186480 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.173917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33117 (local address 127.28.94.62:40993)
0806 04:30:03.174485 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.174508 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.174914 (+ 406us) server_negotiation.cc:407] Connection header received
0806 04:30:03.176559 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.176571 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.176939 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.177293 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.178822 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.179865 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.181177 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.181719 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.181895 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.185635 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:33117: BlockingRecv error: recv got EOF from 127.28.94.1:33117 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":124,"threads_started":1}
I20260806 04:30:03.186487 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.173981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:33117)
0806 04:30:03.174703 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.174758 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.176084 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.177432 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.177441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.177991 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.178626 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.178644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.180033 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.180038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.181037 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.181051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.181920 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.181926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.182136 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.182826 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.182846 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.185345 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":142,"threads_started":1}
W20260806 04:30:03.187292 8659 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:40993 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.94.62:40993: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.193029 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.187817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:36715)
0806 04:30:03.187929 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.187943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.188035 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.188264 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.188266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.188436 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.188553 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.188559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.189374 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.189377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.189937 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.189946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.190396 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.190399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.190495 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.190973 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.190985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.192829 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 04:30:03.193334 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.187923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:36715 (local address 127.28.94.62:40993)
0806 04:30:03.188045 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.188049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.188065 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:03.188114 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.188116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.188165 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.188232 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.188671 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.189264 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.190059 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.190286 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.190377 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.193060 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:36715: BlockingRecv error: recv got EOF from 127.28.94.1:36715 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.198560 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.193652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:51433)
0806 04:30:03.193761 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.193775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.193855 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.194084 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.194087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.194264 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.194351 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.194356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.195120 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.195123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.195651 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.195659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.196087 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.196090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.196166 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.196630 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.196642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.198403 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 04:30:03.198683 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.193729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:51433 (local address 127.28.94.62:40993)
0806 04:30:03.193886 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.193890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.193904 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:30:03.193943 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.193945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.193988 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.194070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.194469 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.195008 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.195760 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.195984 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.196082 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.198561 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:51433: BlockingRecv error: recv got EOF from 127.28.94.1:51433 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":145,"threads_started":1}
W20260806 04:30:03.198899 8659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:30:03.299726 8661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:03.300036 8661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:03.300081 8661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:03.304356 8661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:03.304450 8661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:03.304514 8661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:30:03.310434 8661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.2.keytab
--principal=oryx/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:43443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=39101
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:03.311668 8661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:03.312785 8661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:03.323158 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:03.323185 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:03.323163 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:03.323551 8661 server_base.cc:1034] running on GCE node
I20260806 04:30:03.324021 8661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:03.324785 8661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:03.325953 8661 hybrid_clock.cc:648] HybridClock initialized: now 1785990603325910 us; error 53 us; skew 500 ppm
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990603, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:03.328967 8661 init.cc:377] Logged in from keytab as oryx/127.28.94.2@KRBTEST.COM (short username oryx)
I20260806 04:30:03.330044 8661 webserver.cc:460] Webserver started at http://127.28.94.2:39101/ using document root <none> and password file <none>
I20260806 04:30:03.330681 8661 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:03.330767 8661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:03.336174 8661 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:03.338850 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.340438 8661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:30:03.340582 8661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "bfe3e15bbbb64c729d7f50b89906db5d"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "bc58644410269e00ec1b7f3d9df6c30b"
server_key_iv: "d60e520e54a5643deaf289a4ccf1fb0a"
server_key_version: "encryptionkey@0"
I20260806 04:30:03.341295 8661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:03.378237 8661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:03.382422 8661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:03.382714 8661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:03.383534 8661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:03.384724 8661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:03.384796 8661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.384863 8661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:03.384922 8661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 04:30:03.396736 8661 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43443
I20260806 04:30:03.396764 8820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43443 every 8 connection(s)
I20260806 04:30:03.398310 8661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:30:03.403733 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8661
I20260806 04:30:03.404284 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:34999
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:40993
--webserver_port=34313
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:30:03.412530 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.400614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:52939 (local address 127.28.94.62:40993)
0806 04:30:03.400773 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.400777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.402083 (+ 1306us) server_negotiation.cc:407] Connection header received
0806 04:30:03.403566 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.403570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.403641 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.403714 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.405235 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.405776 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.407323 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.407733 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.407879 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.412368 (+ 4489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:52939: BlockingRecv error: recv got EOF from 127.28.94.2:52939 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:30:03.413640 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.401271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:52939)
0806 04:30:03.401900 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.401952 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.403291 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.403957 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.403966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.404372 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.405066 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.405085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.405914 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.405917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.407163 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.407178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.407938 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.407943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.408174 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.408839 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.408863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.412148 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":176,"threads_started":1}
W20260806 04:30:03.414363 8821 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:40993 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.94.62:40993: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.421787 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.414947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:44665)
0806 04:30:03.415161 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.415177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.415270 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.415579 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.415582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.415782 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.415923 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.415931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.417044 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.417048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.417929 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.417956 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.418596 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.418600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.418704 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.419284 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.419300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.421616 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260806 04:30:03.421939 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.415065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:44665 (local address 127.28.94.62:40993)
0806 04:30:03.415218 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.415223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.415242 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:30:03.415380 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.415384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.415445 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.415546 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.416059 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.416873 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.418087 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.418465 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.418575 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.421803 (+ 3228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:44665: BlockingRecv error: recv got EOF from 127.28.94.2:44665 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.429320 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.422680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:41165)
0806 04:30:03.422992 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.423007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.423124 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.423382 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.423385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.423558 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.423672 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.423679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.424718 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.424722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.425632 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.425646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.426242 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.426245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.426334 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.426882 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.426899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.429154 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235}
W20260806 04:30:03.429507 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.422769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:41165 (local address 127.28.94.62:40993)
0806 04:30:03.422896 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.422900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.423093 (+ 193us) server_negotiation.cc:407] Connection header received
0806 04:30:03.423217 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.423220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.423274 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.423352 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.423785 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.424591 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.425756 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.426111 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.426223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.429356 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:41165: BlockingRecv error: recv got EOF from 127.28.94.2:41165 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 04:30:03.429890 8821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:30:03.534366 8826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:03.534688 8826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:03.534751 8826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:03.539047 8826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:03.539144 8826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:03.539232 8826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:30:03.545287 8826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/krb5kdc/oryx_127.28.94.3.keytab
--principal=oryx/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:34999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=34313
--tserver_master_addrs=127.28.94.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:03.546599 8826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:03.547771 8826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:30:03.558614 8826 server_base.cc:1034] running on GCE node
W20260806 04:30:03.558576 8831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:03.558553 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:03.558552 8832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:03.559365 8826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:03.560102 8826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:03.561285 8826 hybrid_clock.cc:648] HybridClock initialized: now 1785990603561259 us; error 35 us; skew 500 ppm
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990603, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:03.564162 8826 init.cc:377] Logged in from keytab as oryx/127.28.94.3@KRBTEST.COM (short username oryx)
I20260806 04:30:03.565219 8826 webserver.cc:460] Webserver started at http://127.28.94.3:34313/ using document root <none> and password file <none>
I20260806 04:30:03.565836 8826 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:03.565927 8826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:03.570288 8826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:30:03.572599 8846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.573982 8826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:03.574127 8826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "d3072c01e45448aebb3fae612c7e723a"
format_stamp: "Formatted at 2026-08-06 04:30:02 on dist-test-slave-ncp9"
server_key: "6adc3baa350167a2559ebbc34a904d43"
server_key_iv: "288793451f3b6ce627e8c8ebfa5b85d6"
server_key_version: "encryptionkey@0"
I20260806 04:30:03.574718 8826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:03.594271 8826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:03.597400 8826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:03.597613 8826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:03.598312 8826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:03.599260 8826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:03.599314 8826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.599373 8826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:03.599402 8826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:03.608505 8826 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:34999
I20260806 04:30:03.608554 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:34999 every 8 connection(s)
I20260806 04:30:03.609717 8826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:30:03.614351 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 8826
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:30:03.622423 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.611657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:45485 (local address 127.28.94.62:40993)
0806 04:30:03.611833 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.611838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.612744 (+ 906us) server_negotiation.cc:407] Connection header received
0806 04:30:03.614161 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.614165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.614229 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.614325 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.615935 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.616716 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.618158 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.618441 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.618520 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.622275 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:45485: BlockingRecv error: recv got EOF from 127.28.94.3:45485 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260806 04:30:03.623313 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.612070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:45485)
0806 04:30:03.612562 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.612634 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.613929 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.614511 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.614521 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.615029 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.615772 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.615795 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.616849 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.616852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.618030 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.618042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.618608 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.618645 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.618846 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.619420 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.619438 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.622147 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
W20260806 04:30:03.623860 8990 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:40993 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.94.62:40993: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
I20260806 04:30:03.624598 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.615424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.28.94.62:40993)
0806 04:30:03.615573 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.615588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.615606 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:30:03.615649 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.615651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.615699 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.615792 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.616629 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.617393 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.618267 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.618562 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.618664 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.620100 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:03.620107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:03.620134 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:03.620166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:03.622232 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:03.622739 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:03.622746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:03.622752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:03.622917 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:03.623152 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:03.623155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:03.623157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:03.623680 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:03.623875 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:03.624196 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 04:30:03.624464 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":55}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.629403 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.624323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:43449)
0806 04:30:03.624503 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.624517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.624604 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.624837 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.624839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.625030 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.625165 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.625172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.625999 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.626003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.626580 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.626591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.627055 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.627059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.627153 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.627621 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.627634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.629260 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260806 04:30:03.629506 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.624398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:43449 (local address 127.28.94.62:40993)
0806 04:30:03.624505 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.624510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.624557 (+ 47us) server_negotiation.cc:407] Connection header received
0806 04:30:03.624680 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.624683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.624730 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.624802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.625268 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.625890 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.626683 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.626947 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.627029 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.629402 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:43449: BlockingRecv error: recv got EOF from 127.28.94.3:43449 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:03 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:03.634662 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:03.630063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:32893)
0806 04:30:03.630172 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:03.630184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:03.630259 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:03.630511 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:03.630514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:03.630661 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:03.630750 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.630755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.631462 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.631465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.631987 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:03.631994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.632410 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.632414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:03.632490 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.632926 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:03.632942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:03.634515 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 04:30:03.634799 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:03.630163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:32893 (local address 127.28.94.62:40993)
0806 04:30:03.630331 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:03.630335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:03.630346 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:30:03.630382 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:03.630384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:03.630419 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:03.630482 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:03.630858 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.631362 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.632092 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:03.632311 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:03.632399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:03.634693 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:32893: BlockingRecv error: recv got EOF from 127.28.94.3:32893 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 04:30:03.634981 8990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:04.205389 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:04.199697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:50735)
0806 04:30:04.199987 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:04.200002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:04.200110 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:04.200366 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:04.200368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:04.200555 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:04.200684 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.200690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.201620 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.201624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.202237 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.202248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.202737 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.202741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.202838 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.203346 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:04.203357 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:04.205250 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
W20260806 04:30:04.205559 8998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:04.199849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:50735 (local address 127.28.94.62:40993)
0806 04:30:04.200131 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:04.200135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:04.200150 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:04.200197 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:04.200199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:04.200244 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:04.200340 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:04.200782 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.201461 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.202364 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.202613 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.202707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.205420 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:50735: BlockingRecv error: recv got EOF from 127.28.94.1:50735 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:04.436070 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:04.430695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:55395)
0806 04:30:04.430955 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:04.430971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:04.431075 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:04.431359 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:04.431361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:04.431543 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:04.431670 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.431676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.432370 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.432372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.432975 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.432982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.433424 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.433427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.433511 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.434041 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:04.434054 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:04.435921 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
W20260806 04:30:04.436216 8998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:04.430732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:55395 (local address 127.28.94.62:40993)
0806 04:30:04.430905 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:04.430909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:04.431041 (+ 132us) server_negotiation.cc:407] Connection header received
0806 04:30:04.431198 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:04.431201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:04.431258 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:04.431325 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:04.431763 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.432288 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.433087 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.433319 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.433400 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.436088 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:55395: BlockingRecv error: recv got EOF from 127.28.94.2:55395 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:04 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:04.643069 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:04.635626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:36553)
0806 04:30:04.635985 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:04.636000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:04.636144 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:04.636460 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:04.636464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:04.636686 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:04.636818 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.636826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.637951 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.637955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.638910 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:04.638926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.639543 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.639547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:04.639646 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.640284 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:04.640300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:04.642847 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":87,"threads_started":1}
W20260806 04:30:04.643162 8998 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:04.635694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:36553 (local address 127.28.94.62:40993)
0806 04:30:04.635828 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:04.635833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:04.636090 (+ 257us) server_negotiation.cc:407] Connection header received
0806 04:30:04.636247 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:04.636250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:04.636333 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:04.636429 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:04.636975 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.637798 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.639067 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:04.639402 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:04.639518 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:04.643033 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:36553: BlockingRecv error: recv got EOF from 127.28.94.3:36553 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:05.211913 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:05.206288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:36925)
0806 04:30:05.206582 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:05.206596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:05.206692 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:05.206912 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:05.206914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:05.207134 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:05.207261 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.207268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.208114 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.208117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.208834 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.208850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.209371 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.209374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.209448 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.209984 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:05.209997 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:05.211760 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
W20260806 04:30:05.212049 9003 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:05.206388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:36925 (local address 127.28.94.62:40993)
0806 04:30:05.206685 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:05.206689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:05.206704 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:05.206752 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:05.206755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:05.206798 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:05.206878 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:05.207370 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.207988 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.208979 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.209233 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.209320 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.211921 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:36925: BlockingRecv error: recv got EOF from 127.28.94.1:36925 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:05.442678 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:05.437082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:34063)
0806 04:30:05.437404 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:05.437418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:05.437538 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:05.437793 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:05.437796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:05.437982 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:05.438089 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.438094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.438818 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.438820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.439378 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.439385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.439795 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.439799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.439870 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.440429 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:05.440445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:05.442492 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
W20260806 04:30:05.442763 9003 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:05.437201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:34063 (local address 127.28.94.62:40993)
0806 04:30:05.437352 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:05.437357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:05.437491 (+ 134us) server_negotiation.cc:407] Connection header received
0806 04:30:05.437623 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:05.437625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:05.437685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:05.437768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:05.438186 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.438725 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.439462 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.439692 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.439764 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.442658 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:34063: BlockingRecv error: recv got EOF from 127.28.94.2:34063 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:05 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:05.650180 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:05.644155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:56389)
0806 04:30:05.644446 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:05.644460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:05.644588 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:05.645001 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:05.645004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:05.645310 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:05.645435 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.645444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.646449 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.646452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.647099 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:05.647111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.647605 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.647608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:05.647672 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.648171 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:05.648184 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:05.650015 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
W20260806 04:30:05.650291 9003 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:05.644352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:56389 (local address 127.28.94.62:40993)
0806 04:30:05.644535 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:05.644539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:05.644558 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:30:05.644711 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:05.644714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:05.644776 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:05.644887 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:05.645562 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.646312 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.647241 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:05.647475 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:05.647570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:05.650178 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:56389: BlockingRecv error: recv got EOF from 127.28.94.3:56389 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:06.219547 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:06.212922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:34875)
0806 04:30:06.213241 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:06.213256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:06.213368 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:06.213622 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:06.213624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:06.213815 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:06.213921 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.213928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.214671 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.214674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.215371 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.215387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.215934 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.215937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.216014 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.216705 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:06.216726 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:06.219379 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
W20260806 04:30:06.219682 9009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:06.213050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:34875 (local address 127.28.94.62:40993)
0806 04:30:06.213333 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:06.213337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:06.213354 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:30:06.213456 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:06.213458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:06.213508 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:06.213588 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:06.214021 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.214568 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.215513 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.215782 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.215872 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.219554 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:34875: BlockingRecv error: recv got EOF from 127.28.94.1:34875 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:06.449590 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:06.443697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:32813)
0806 04:30:06.444012 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:06.444026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:06.444153 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:06.444421 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:06.444424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:06.444642 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:06.444758 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.444765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.445716 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.445721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.446438 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.446450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.446911 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.446915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.446997 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.447535 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:06.447548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:06.449425 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
W20260806 04:30:06.449713 9009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:06.443783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:32813 (local address 127.28.94.62:40993)
0806 04:30:06.443938 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:06.443944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:06.444105 (+ 161us) server_negotiation.cc:407] Connection header received
0806 04:30:06.444255 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:06.444258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:06.444305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:06.444375 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:06.444916 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.445593 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.446558 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.446801 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.446892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.449603 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:32813: BlockingRecv error: recv got EOF from 127.28.94.2:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:06 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:06.656430 9011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:06.651017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:53335)
0806 04:30:06.651320 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:06.651334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:06.651443 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:06.651670 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:06.651672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:06.651850 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:06.651953 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.651959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.652683 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.652685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.653316 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:06.653326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.653771 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.653774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:06.653839 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.654335 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:06.654347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:06.656270 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
W20260806 04:30:06.656533 9009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:06.651110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:53335 (local address 127.28.94.62:40993)
0806 04:30:06.651255 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:06.651260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:06.651436 (+ 176us) server_negotiation.cc:407] Connection header received
0806 04:30:06.651514 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:06.651517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:06.651568 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:06.651653 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:06.652056 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.652589 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.653425 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:06.653669 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:06.653758 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:06.656426 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:53335: BlockingRecv error: recv got EOF from 127.28.94.3:53335 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:07.229228 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:07.220563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:47559)
0806 04:30:07.220873 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:07.220919 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:07.221037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:07.221293 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:07.221295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:07.221493 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:07.221595 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.221628 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.222395 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.222399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.222985 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.222995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.223512 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.223515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.223589 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.227032 (+ 3443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:07.227049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:07.229086 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
W20260806 04:30:07.229420 9013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:07.220740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:47559 (local address 127.28.94.62:40993)
0806 04:30:07.220995 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:07.220999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:07.221014 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:07.221123 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:07.221125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:07.221184 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:07.221262 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:07.221721 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.222287 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.223110 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.223360 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.223444 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.229269 (+ 5825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:47559: BlockingRecv error: recv got EOF from 127.28.94.1:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:07.456228 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:07.450512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:45223)
0806 04:30:07.450881 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:07.450898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:07.451043 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:07.451375 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:07.451378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:07.451604 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:07.451723 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.451729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.452520 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.452523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.453226 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.453236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.453691 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.453694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.453756 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.454240 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:07.454250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:07.456079 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
W20260806 04:30:07.456405 9013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:07.450574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:45223 (local address 127.28.94.62:40993)
0806 04:30:07.450718 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:07.450722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:07.450971 (+ 249us) server_negotiation.cc:407] Connection header received
0806 04:30:07.451176 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:07.451180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:07.451241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:07.451305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:07.451841 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.452401 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.453342 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.453569 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.453645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.456283 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:45223: BlockingRecv error: recv got EOF from 127.28.94.2:45223 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:07 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:07.663182 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:07.657502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:44221)
0806 04:30:07.657839 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:07.657853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:07.657962 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:07.658205 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:07.658208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:07.658393 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:07.658496 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.658502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.659217 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.659219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.659783 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:07.659791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.660311 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.660315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:07.660408 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.660981 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:07.660997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:07.663038 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
W20260806 04:30:07.663340 9013 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:07.657609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:44221 (local address 127.28.94.62:40993)
0806 04:30:07.657772 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:07.657777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:07.657922 (+ 145us) server_negotiation.cc:407] Connection header received
0806 04:30:07.658040 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:07.658043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:07.658096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:07.658175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:07.658585 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.659119 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.659889 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:07.660171 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:07.660268 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:07.663217 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:44221: BlockingRecv error: recv got EOF from 127.28.94.3:44221 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:08.235725 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:08.230097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:59331)
0806 04:30:08.230405 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:08.230421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:08.230541 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:08.230883 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:08.230887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:08.231083 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:08.231191 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.231199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.231995 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.231999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.232605 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.232618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.233092 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.233096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.233170 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.233699 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:08.233710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:08.235563 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
W20260806 04:30:08.235930 9018 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:08.230165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:59331 (local address 127.28.94.62:40993)
0806 04:30:08.230410 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:08.230415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:08.230496 (+ 81us) server_negotiation.cc:407] Connection header received
0806 04:30:08.230649 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:08.230652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:08.230713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:08.230800 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:08.231322 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.231887 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.232717 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.232974 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.233058 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.235777 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:59331: BlockingRecv error: recv got EOF from 127.28.94.1:59331 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:08.463186 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:08.457527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:43309)
0806 04:30:08.457842 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:08.457857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:08.457973 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:08.458227 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:08.458231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:08.458475 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:08.458614 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.458621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.459439 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.459443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.460033 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.460042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.460485 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.460488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.460583 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.461088 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:08.461100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:08.463042 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
W20260806 04:30:08.463344 9018 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:08.457824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:43309 (local address 127.28.94.62:40993)
0806 04:30:08.457993 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:08.457998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:08.458016 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:30:08.458070 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:08.458074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:08.458120 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:08.458189 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:08.458737 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.459325 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.460152 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.460380 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.460467 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.463212 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:43309: BlockingRecv error: recv got EOF from 127.28.94.2:43309 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:08 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:08.669581 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:08.664137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:34181)
0806 04:30:08.664461 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:08.664474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:08.664579 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:08.664797 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:08.664799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:08.665012 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:08.665122 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.665128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.665820 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.665822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.666410 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:08.666419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.666919 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.666923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:08.667007 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.667513 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:08.667525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:08.669451 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
W20260806 04:30:08.669754 9018 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:08.664249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:34181 (local address 127.28.94.62:40993)
0806 04:30:08.664403 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:08.664408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:08.664539 (+ 131us) server_negotiation.cc:407] Connection header received
0806 04:30:08.664655 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:08.664657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:08.664703 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:08.664768 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:08.665209 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.665728 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.666567 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:08.666804 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:08.666898 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:08.669628 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:34181: BlockingRecv error: recv got EOF from 127.28.94.3:34181 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:09.242652 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:09.236694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:50475)
0806 04:30:09.237093 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:09.237108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:09.237213 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:09.237477 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:09.237479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:09.237722 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:09.237856 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.237863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.238728 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.238732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.239411 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.239422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.239890 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.239894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.239974 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.240483 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:09.240495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:09.242479 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
W20260806 04:30:09.242826 9023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:09.236873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:50475 (local address 127.28.94.62:40993)
0806 04:30:09.237147 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:09.237151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:09.237167 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:09.237303 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:09.237305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:09.237354 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:09.237429 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:09.237997 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.238606 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.239512 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.239753 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.239836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.242694 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:50475: BlockingRecv error: recv got EOF from 127.28.94.1:50475 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":66,"threads_started":1}
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:09.469645 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:09.464055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:46211)
0806 04:30:09.464355 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:09.464371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:09.464493 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:09.464753 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:09.464757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:09.464999 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:09.465131 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.465139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.465904 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.465907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.466528 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.466537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.466991 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.466994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.467076 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.467575 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:09.467588 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:09.469504 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
W20260806 04:30:09.469818 9023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:09.464112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46211 (local address 127.28.94.62:40993)
0806 04:30:09.464255 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:09.464259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:09.464443 (+ 184us) server_negotiation.cc:407] Connection header received
0806 04:30:09.464579 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:09.464582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:09.464631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:09.464709 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:09.465244 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.465794 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.466647 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.466885 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.466974 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.469684 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:46211: BlockingRecv error: recv got EOF from 127.28.94.2:46211 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:09 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:09.675935 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:09.670528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:44311)
0806 04:30:09.670860 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:09.670873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:09.671005 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:09.671281 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:09.671283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:09.671533 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:09.671653 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.671661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.672424 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.672427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.673094 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:09.673105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.673483 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.673486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:09.673544 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.674035 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:09.674046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:09.675802 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
W20260806 04:30:09.676074 9023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:09.670627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:44311 (local address 127.28.94.62:40993)
0806 04:30:09.670762 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:09.670767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:09.670939 (+ 172us) server_negotiation.cc:407] Connection header received
0806 04:30:09.671098 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:09.671100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:09.671164 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:09.671245 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:09.671775 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.672327 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.673189 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:09.673403 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:09.673472 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:09.675965 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:44311: BlockingRecv error: recv got EOF from 127.28.94.3:44311 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:10.249449 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:10.243563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:39661)
0806 04:30:10.243951 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:10.243966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:10.244087 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:10.244320 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:10.244322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:10.244526 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:10.244699 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.244714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.245528 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.245530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.246202 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.246214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.246725 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.246727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.246788 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.247300 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:10.247311 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:10.249274 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":102,"threads_started":1}
W20260806 04:30:10.249611 9027 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:10.243694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:39661 (local address 127.28.94.62:40993)
0806 04:30:10.243955 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:10.243959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:10.244046 (+ 87us) server_negotiation.cc:407] Connection header received
0806 04:30:10.244164 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:10.244166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:10.244215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:10.244295 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:10.244830 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.245426 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.246325 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.246624 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.246711 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.249474 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:39661: BlockingRecv error: recv got EOF from 127.28.94.1:39661 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":65,"threads_started":1}
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:10.479151 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:10.470813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:60055)
0806 04:30:10.471342 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:10.471364 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:10.471550 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:10.471955 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:10.471959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:10.472288 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:10.472434 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.472442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.473559 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.473564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.474323 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.474340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.474988 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.475005 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.475140 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.475906 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:10.475926 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:10.478854 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":135,"threads_started":1}
W20260806 04:30:10.479274 9027 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:10.470983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:60055 (local address 127.28.94.62:40993)
0806 04:30:10.471254 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:10.471268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:10.471471 (+ 203us) server_negotiation.cc:407] Connection header received
0806 04:30:10.471712 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:10.471718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:10.471801 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:10.471900 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:10.472629 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.473402 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.474521 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.474827 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.474972 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.479111 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:60055: BlockingRecv error: recv got EOF from 127.28.94.2:60055 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:10 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:10.685194 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:10.677207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:39797)
0806 04:30:10.677654 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:10.677674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:10.677816 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:10.678192 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:10.678194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:10.678497 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:10.678636 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.678645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.679675 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.679682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.680451 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:10.680468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.681122 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.681128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:10.681237 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.681940 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:10.681960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:10.685020 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":134,"threads_started":1}
W20260806 04:30:10.685314 9027 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:10.677324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:39797 (local address 127.28.94.62:40993)
0806 04:30:10.677569 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:10.677574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:10.677763 (+ 189us) server_negotiation.cc:407] Connection header received
0806 04:30:10.677960 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:10.677970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:10.678045 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:10.678134 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:10.678815 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.679541 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.680604 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:10.680975 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:10.681095 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:10.685184 (+ 4089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:39797: BlockingRecv error: recv got EOF from 127.28.94.3:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:11.259371 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:11.250597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:33885)
0806 04:30:11.251140 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:11.251169 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:11.251328 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:11.251792 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:11.251804 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:11.252133 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:11.252340 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.252353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.253506 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.253509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.254448 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.254469 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.255230 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.255234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.255343 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.256069 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:11.256090 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:11.258982 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":126,"threads_started":1}
W20260806 04:30:11.259577 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:11.250896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33885 (local address 127.28.94.62:40993)
0806 04:30:11.251319 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:11.251325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:11.251340 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:11.251438 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:11.251468 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:11.251552 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:11.251719 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:11.252566 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.253344 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.254670 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.255059 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.255190 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.259342 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:33885: BlockingRecv error: recv got EOF from 127.28.94.1:33885 (error 108)
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":159,"threads_started":1}
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:11.486057 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:11.480247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:41287)
0806 04:30:11.480556 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:11.480574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:11.480715 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:11.481040 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:11.481042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:11.481248 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:11.481383 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.481391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.482452 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.482455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.483022 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.483031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.483476 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.483479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.483539 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.484092 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:11.484106 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:11.485925 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
W20260806 04:30:11.486315 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:11.480393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:41287 (local address 127.28.94.62:40993)
0806 04:30:11.480556 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:11.480562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:11.480662 (+ 100us) server_negotiation.cc:407] Connection header received
0806 04:30:11.480817 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:11.480820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:11.480882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:11.480995 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:11.481512 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.482343 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.483146 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.483382 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.483472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.486186 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:41287: BlockingRecv error: recv got EOF from 127.28.94.2:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:11 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:11.691673 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:11.686314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:47483)
0806 04:30:11.686603 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:11.686616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:11.686717 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:11.686932 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:11.686934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:11.687120 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:11.687219 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.687224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.688042 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.688046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.688640 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:11.688650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.689184 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.689187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:11.689257 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.689747 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:11.689759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:11.691516 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
W20260806 04:30:11.691831 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:11.686362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:47483 (local address 127.28.94.62:40993)
0806 04:30:11.686489 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:11.686492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:11.686677 (+ 185us) server_negotiation.cc:407] Connection header received
0806 04:30:11.686789 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:11.686791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:11.686842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:11.686908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:11.687370 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.687921 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.688775 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:11.689068 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:11.689162 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:11.691708 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:47483: BlockingRecv error: recv got EOF from 127.28.94.3:47483 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:12.269936 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:12.260735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:60697)
0806 04:30:12.261083 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:12.261097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:12.261218 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:12.261460 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:12.261462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:12.261653 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:12.261753 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.261759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.262499 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.262502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.263147 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.263160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.263665 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.263668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.263736 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.267727 (+ 3991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:12.267744 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:12.269794 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":153,"threads_started":1}
W20260806 04:30:12.270159 9035 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:12.260814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:60697 (local address 127.28.94.62:40993)
0806 04:30:12.261078 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:12.261081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:12.261177 (+ 96us) server_negotiation.cc:407] Connection header received
0806 04:30:12.261298 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:12.261300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:12.261347 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:12.261433 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:12.261848 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.262406 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.263262 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.263544 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.263649 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.269995 (+ 6346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:60697: BlockingRecv error: recv got EOF from 127.28.94.1:60697 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:12.492766 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:12.487118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:49323)
0806 04:30:12.487446 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:12.487462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:12.487585 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:12.487923 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:12.487926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:12.488122 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:12.488244 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.488250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.489057 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.489060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.489631 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.489639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.490060 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.490063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.490122 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.490610 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:12.490621 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:12.492636 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
W20260806 04:30:12.492931 9035 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:12.487252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:49323 (local address 127.28.94.62:40993)
0806 04:30:12.487423 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:12.487428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:12.487535 (+ 107us) server_negotiation.cc:407] Connection header received
0806 04:30:12.487669 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:12.487672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:12.487721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:12.487792 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:12.488371 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.488956 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.489739 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.489967 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.490048 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.492781 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:49323: BlockingRecv error: recv got EOF from 127.28.94.2:49323 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:12 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:12.698393 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:12.692671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:34827)
0806 04:30:12.693029 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:12.693044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:12.693148 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:12.693391 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:12.693393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:12.693582 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:12.693683 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.693688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.694541 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.694545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.695278 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:12.695292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.695750 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.695755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:12.695844 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.696329 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:12.696340 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:12.698263 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
W20260806 04:30:12.698541 9035 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:12.692871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:34827 (local address 127.28.94.62:40993)
0806 04:30:12.693069 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:12.693073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:12.693097 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:30:12.693238 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:12.693241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:12.693291 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:12.693361 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:12.693821 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.694411 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.695406 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:12.695649 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:12.695730 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:12.698424 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:34827: BlockingRecv error: recv got EOF from 127.28.94.3:34827 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:13.276289 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:13.270886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:50567)
0806 04:30:13.271197 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:13.271211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:13.271302 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:13.271507 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:13.271509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:13.271689 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:13.271787 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.271793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.272506 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.272509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.273163 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.273174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.273598 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.273601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.273661 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.274240 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:13.274251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:13.276133 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
W20260806 04:30:13.276465 9039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:13.271043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:50567 (local address 127.28.94.62:40993)
0806 04:30:13.271290 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:13.271293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:13.271307 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:30:13.271383 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:13.271385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:13.271430 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:13.271495 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:13.271880 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.272423 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.273270 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.273506 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.273588 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.276320 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:50567: BlockingRecv error: recv got EOF from 127.28.94.1:50567 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:13.499280 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:13.493734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:34257)
0806 04:30:13.494064 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:13.494080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:13.494192 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:13.494426 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:13.494428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:13.494629 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:13.494738 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.494745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.495476 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.495479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.496057 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.496066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.496458 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.496461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.496523 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.497123 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:13.497136 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:13.499152 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
W20260806 04:30:13.499531 9039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:13.493900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:34257 (local address 127.28.94.62:40993)
0806 04:30:13.494089 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:13.494092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:13.494153 (+ 61us) server_negotiation.cc:407] Connection header received
0806 04:30:13.494274 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:13.494276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:13.494323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:13.494394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:13.494843 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.495381 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.496151 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.496370 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.496439 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.499312 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:34257: BlockingRecv error: recv got EOF from 127.28.94.2:34257 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:13 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:13.705201 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:13.699392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:51371)
0806 04:30:13.699735 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:13.699754 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:13.699898 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:13.700194 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:13.700197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:13.700412 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:13.700548 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.700557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.701562 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.701565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.702129 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:13.702137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.702518 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.702520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:13.702578 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.703117 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:13.703131 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:13.705005 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
W20260806 04:30:13.705282 9039 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:13.699610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:51371 (local address 127.28.94.62:40993)
0806 04:30:13.699767 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:13.699771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:13.699835 (+ 64us) server_negotiation.cc:407] Connection header received
0806 04:30:13.700002 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:13.700005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:13.700069 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:13.700139 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:13.700707 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.701456 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.702229 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:13.702444 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:13.702511 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:13.705173 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:51371: BlockingRecv error: recv got EOF from 127.28.94.3:51371 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:14.282588 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:14.277331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:45957)
0806 04:30:14.277642 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:14.277656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:14.277761 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:14.277992 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:14.277994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:14.278186 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:14.278289 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.278294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.279124 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.279127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.279692 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.279702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.280096 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.280099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.280161 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.280680 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:14.280692 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:14.282430 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
W20260806 04:30:14.282775 9044 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:14.277415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:45957 (local address 127.28.94.62:40993)
0806 04:30:14.277732 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:14.277737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:14.277753 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:14.277846 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:14.277849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:14.277898 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:14.277967 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:14.278380 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.279016 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.279794 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.280012 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.280080 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.282626 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:45957: BlockingRecv error: recv got EOF from 127.28.94.1:45957 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:14.509953 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:14.500056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:46915)
0806 04:30:14.500364 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:14.500378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:14.500496 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:14.500800 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:14.500803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:14.501043 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:14.501151 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.501158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.501912 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.501916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.502496 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.502505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.502960 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.502964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.503043 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.507606 (+ 4563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:14.507645 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:14.509776 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20260806 04:30:14.510092 9044 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:14.500134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46915 (local address 127.28.94.62:40993)
0806 04:30:14.500277 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:14.500282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:14.500452 (+ 170us) server_negotiation.cc:407] Connection header received
0806 04:30:14.500635 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:14.500638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:14.500689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:14.500755 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:14.501274 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.501803 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.502620 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.502848 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.502938 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.509980 (+ 7042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:46915: BlockingRecv error: recv got EOF from 127.28.94.2:46915 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:14 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:14.712239 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:14.706126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:37757)
0806 04:30:14.706487 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:14.706505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:14.706618 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:14.707004 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:14.707007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:14.707204 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:14.707311 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.707317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.708107 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.708110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.708736 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:14.708746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.709333 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.709337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:14.709411 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.709962 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:14.709976 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:14.711988 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
W20260806 04:30:14.712344 9044 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:14.706249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:37757 (local address 127.28.94.62:40993)
0806 04:30:14.706530 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:14.706535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:14.706556 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:30:14.706806 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:14.706810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:14.706866 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:14.706940 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:14.707418 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.707991 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.708951 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:14.709191 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:14.709284 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:14.712229 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:37757: BlockingRecv error: recv got EOF from 127.28.94.3:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:15.289067 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:15.283463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:33181)
0806 04:30:15.283760 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:15.283776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:15.283888 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:15.284115 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:15.284117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:15.284308 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:15.284408 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.284413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.285199 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.285201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.285748 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.285755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.286201 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.286204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.286266 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.286811 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:15.286823 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:15.288866 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
W20260806 04:30:15.289145 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:15.283564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33181 (local address 127.28.94.62:40993)
0806 04:30:15.283891 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:15.283895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:15.283911 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:15.283971 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:15.283974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:15.284019 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:15.284090 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:15.284503 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.285107 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.285864 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.286103 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.286185 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.289026 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:33181: BlockingRecv error: recv got EOF from 127.28.94.1:33181 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:15.516983 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:15.510986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:58465)
0806 04:30:15.511271 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:15.511286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:15.511397 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:15.511636 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:15.511638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:15.511850 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:15.511981 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.511991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.513161 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.513164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.513746 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.513755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.514141 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.514143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.514203 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.514683 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:15.514695 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:15.516794 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
W20260806 04:30:15.517088 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:15.511094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:58465 (local address 127.28.94.62:40993)
0806 04:30:15.511284 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:15.511288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:15.511359 (+ 71us) server_negotiation.cc:407] Connection header received
0806 04:30:15.511476 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:15.511479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:15.511527 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:15.511610 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:15.512154 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.513033 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.513837 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.514050 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.514125 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.516967 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:58465: BlockingRecv error: recv got EOF from 127.28.94.2:58465 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:15 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:15.718619 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:15.713171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:60925)
0806 04:30:15.713459 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:15.713473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:15.713582 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:15.713806 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:15.713808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:15.714004 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:15.714112 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.714118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.714865 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.714867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.715433 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:15.715441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.715841 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.715843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:15.715900 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.716450 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:15.716463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:15.718447 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260806 04:30:15.718776 9050 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:15.713263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:60925 (local address 127.28.94.62:40993)
0806 04:30:15.713395 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:15.713399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:15.713529 (+ 130us) server_negotiation.cc:407] Connection header received
0806 04:30:15.713653 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:15.713655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:15.713703 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:15.713775 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:15.714224 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.714773 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.715522 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:15.715754 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:15.715829 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:15.718661 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:60925: BlockingRecv error: recv got EOF from 127.28.94.3:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:16.295498 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:16.290076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:53753)
0806 04:30:16.290379 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:16.290396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:16.290499 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:16.290726 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:16.290728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:16.290922 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:16.291024 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.291030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.291767 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.291769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.292325 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.292332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.292747 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.292749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.292809 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.293389 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:16.293403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:16.295327 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
W20260806 04:30:16.295647 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:16.290170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:53753 (local address 127.28.94.62:40993)
0806 04:30:16.290445 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:16.290448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:16.290464 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:16.290573 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:16.290576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:16.290623 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:16.290698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:16.291117 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.291680 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.292411 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.292659 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.292732 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.295520 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:53753: BlockingRecv error: recv got EOF from 127.28.94.1:53753 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:16.523732 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:16.518090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:43925)
0806 04:30:16.518410 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:16.518425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:16.518525 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:16.518757 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:16.518759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:16.518941 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:16.519043 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.519048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.519744 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.519746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.520301 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.520310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.520680 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.520682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.520738 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.521306 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:16.521319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:16.523550 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
W20260806 04:30:16.523801 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:16.518311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:43925 (local address 127.28.94.62:40993)
0806 04:30:16.518470 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:16.518474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:16.518490 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:30:16.518608 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:16.518610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:16.518661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:16.518729 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:16.519137 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.519658 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.520385 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.520595 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.520669 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.523703 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:43925: BlockingRecv error: recv got EOF from 127.28.94.2:43925 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:16 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:16.725397 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:16.719613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:40677)
0806 04:30:16.719902 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:16.719915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:16.720025 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:16.720252 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:16.720254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:16.720450 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:16.720552 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.720557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.721398 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.721404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.721976 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:16.721984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.722507 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.722511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:16.722583 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.723119 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:16.723131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:16.725230 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
W20260806 04:30:16.725533 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:16.719716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:40677 (local address 127.28.94.62:40993)
0806 04:30:16.719913 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:16.719918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:16.719986 (+ 68us) server_negotiation.cc:407] Connection header received
0806 04:30:16.720101 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:16.720104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:16.720155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:16.720223 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:16.720678 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.721278 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.722109 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:16.722368 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:16.722458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:16.725410 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:40677: BlockingRecv error: recv got EOF from 127.28.94.3:40677 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:17.305228 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:17.296357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:44211)
0806 04:30:17.296638 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:17.296653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:17.296755 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:17.297099 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:17.297102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:17.297307 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:17.297411 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.297417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.298224 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.298227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.298815 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.298823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.299228 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.299231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.299291 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.302654 (+ 3363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:17.302675 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:17.305053 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
W20260806 04:30:17.305356 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:17.296435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:44211 (local address 127.28.94.62:40993)
0806 04:30:17.296801 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:17.296804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:17.296819 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:17.296859 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:17.296861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:17.296946 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:17.297043 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:17.297558 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.298121 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.298911 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.299140 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.299212 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.305231 (+ 6019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:44211: BlockingRecv error: recv got EOF from 127.28.94.1:44211 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":90,"threads_started":1}
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:17.530390 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:17.524752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:57501)
0806 04:30:17.525084 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:17.525101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:17.525233 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:17.525497 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:17.525500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:17.525723 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:17.525827 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.525833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.526604 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.526608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.527179 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.527189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.527679 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.527683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.527766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.528277 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:17.528290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:17.530232 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":127,"threads_started":1}
W20260806 04:30:17.530517 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:17.524939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:57501 (local address 127.28.94.62:40993)
0806 04:30:17.525085 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:17.525091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:17.525177 (+ 86us) server_negotiation.cc:407] Connection header received
0806 04:30:17.525333 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:17.525336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:17.525384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:17.525452 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:17.525950 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.526482 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.527294 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.527560 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.527654 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.530395 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:57501: BlockingRecv error: recv got EOF from 127.28.94.2:57501 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:17 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.3@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:17.731779 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:17.726208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.3:42603)
0806 04:30:17.726528 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:17.726541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:17.726673 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:17.726927 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:17.726930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:17.727150 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:17.727251 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.727257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.728038 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.728041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.728630 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:17.728640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.729143 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.729147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:17.729228 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.729743 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:17.729755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:17.731585 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":130,"threads_started":1}
W20260806 04:30:17.731871 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:17.726311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:42603 (local address 127.28.94.62:40993)
0806 04:30:17.726459 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:17.726464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:17.726615 (+ 151us) server_negotiation.cc:407] Connection header received
0806 04:30:17.726766 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:17.726770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:17.726820 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:17.726886 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:17.727380 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.727921 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.728750 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:17.729028 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:17.729129 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:17.731761 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.3:42603: BlockingRecv error: recv got EOF from 127.28.94.3:42603 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 04:30:18 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:18 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.1@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:18.311909 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:18.306318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.1:54055)
0806 04:30:18.306620 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:18.306636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:18.306727 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:18.306961 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:18.306963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:18.307150 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:18.307272 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:18.307282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.308086 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.308089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:18.308678 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:18.308688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.309184 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.309187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:18.309250 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:18.309841 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:18.309856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:18.311787 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
W20260806 04:30:18.312080 9065 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:18.306419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:54055 (local address 127.28.94.62:40993)
0806 04:30:18.306695 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:18.306700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:18.306719 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:30:18.306811 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:18.306813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:18.306859 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:18.306936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:18.307416 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.307978 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.308789 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.309077 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.309181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:18.311936 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.1:54055: BlockingRecv error: recv got EOF from 127.28.94.1:54055 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
Aug 06 04:30:18 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:30:18 dist-test-slave-ncp9 krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.94.2@KRBTEST.COM for oryx/127.28.94.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:30:18.536964 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:18.531487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:40993 (local address 127.28.94.2:36011)
0806 04:30:18.531793 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:18.531808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:18.531913 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:18.532152 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:18.532154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:18.532337 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:18.532438 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:18.532444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.533204 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.533207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:18.533766 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:18.533773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.534293 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.534297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:18.534380 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:18.534953 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:18.534966 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:18.536813 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.94.62:40993: Server oryx/127.28.94.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
W20260806 04:30:18.537127 9065 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:18.531631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:36011 (local address 127.28.94.62:40993)
0806 04:30:18.531795 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:18.531799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:18.531877 (+ 78us) server_negotiation.cc:407] Connection header received
0806 04:30:18.531990 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:18.531992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:18.532047 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:18.532124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:18.532530 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.533118 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.533874 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.534178 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.534269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:18.536999 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.94.2:36011: BlockingRecv error: recv got EOF from 127.28.94.2:36011 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 04:30:18.616828 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8467
I20260806 04:30:18.632223 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8661
I20260806 04:30:18.650179 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8826
I20260806 04:30:18.659271 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 8375
2026-08-06T04:30:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18322 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T04:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:18Z Disabled control of system clock
I20260806 04:30:18.699440 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38395
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:30:18.821352 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:18.821796 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:18.821892 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:18.826231 9074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:18.826314 9074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:18.826335 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:18.826357 9074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:18.826377 9074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:18.831575 9074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:38395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:18.832932 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:18.834527 9074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:18.843964 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:18.844033 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:18.844013 9074 server_base.cc:1034] running on GCE node
W20260806 04:30:18.844017 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:18.844810 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:18.846158 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:18.847347 9074 hybrid_clock.cc:648] HybridClock initialized: now 1785990618847326 us; error 35 us; skew 500 ppm
I20260806 04:30:18.849525 9074 webserver.cc:460] Webserver started at http://127.28.94.62:34313/ using document root <none> and password file <none>
I20260806 04:30:18.850205 9074 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:18.850279 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:18.850513 9074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:18.852860 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "d2f5ebef94684534b2ded4febaf9fdf0"
format_stamp: "Formatted at 2026-08-06 04:30:18 on dist-test-slave-ncp9"
server_key: "4fc95e24494209eb2f652e0b74235280"
server_key_iv: "61a25f33949a2736b72c152585cbbe29"
server_key_version: "encryptionkey@0"
I20260806 04:30:18.853592 9074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "d2f5ebef94684534b2ded4febaf9fdf0"
format_stamp: "Formatted at 2026-08-06 04:30:18 on dist-test-slave-ncp9"
server_key: "4fc95e24494209eb2f652e0b74235280"
server_key_iv: "61a25f33949a2736b72c152585cbbe29"
server_key_version: "encryptionkey@0"
I20260806 04:30:18.858275 9074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:30:18.861126 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:18.862478 9074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:18.862609 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "d2f5ebef94684534b2ded4febaf9fdf0"
format_stamp: "Formatted at 2026-08-06 04:30:18 on dist-test-slave-ncp9"
server_key: "4fc95e24494209eb2f652e0b74235280"
server_key_iv: "61a25f33949a2736b72c152585cbbe29"
server_key_version: "encryptionkey@0"
I20260806 04:30:18.862748 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:18.882601 9074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:18.883416 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:18.883625 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:18.891185 9074 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:38395
I20260806 04:30:18.891201 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:38395 every 8 connection(s)
I20260806 04:30:18.892237 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:30:18.895757 9155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:18.896868 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9074
I20260806 04:30:18.897033 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:30:18.897341 29048 external_mini_cluster.cc:1428] Setting key 65e3740e636823c1054f04215e0978aa
I20260806 04:30:18.902810 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0: Bootstrap starting.
I20260806 04:30:18.905411 9155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:18.906226 9155 log.cc:824] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:18.907390 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:18.898906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.28.94.62:38395)
0806 04:30:18.899682 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:18.899705 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:18.899755 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:30:18.900657 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:18.900678 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:18.901183 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:18.901498 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:18.901870 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.902864 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.903793 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:18.904302 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:18.904474 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:18.905350 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:18.905356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:18.905371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:18.905828 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:18.905887 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:18.905909 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:18.906015 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:18.906153 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 04:30:18.906454 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":137,"threads_started":1}
I20260806 04:30:18.908545 9155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0: No bootstrap required, opened a new log
I20260806 04:30:18.912444 9155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } }
I20260806 04:30:18.912705 9155 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:18.912753 9155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f5ebef94684534b2ded4febaf9fdf0, State: Initialized, Role: FOLLOWER
I20260806 04:30:18.913255 9155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } }
I20260806 04:30:18.913391 9155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:30:18.913456 9155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:30:18.913558 9155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:18.914669 9155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } }
I20260806 04:30:18.915059 9155 leader_election.cc:302] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f5ebef94684534b2ded4febaf9fdf0; no voters:
I20260806 04:30:18.915503 9155 leader_election.cc:288] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:30:18.915607 9161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:30:18.915814 9161 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [term 1 LEADER]: Becoming Leader. State: Replica: d2f5ebef94684534b2ded4febaf9fdf0, State: Running, Role: LEADER
I20260806 04:30:18.916213 9161 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } }
I20260806 04:30:18.917054 9155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:30:18.918164 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f5ebef94684534b2ded4febaf9fdf0. Latest consensus state: current_term: 1 leader_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } } }
I20260806 04:30:18.918294 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:18.918138 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f5ebef94684534b2ded4febaf9fdf0" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38395 } } }
I20260806 04:30:18.918433 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:18.918742 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:30:18.922394 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:30:18.928757 9170 catalog_manager.cc:1347] Generated new cluster ID: a7d5f99ca1d5471981dc55e7923489f7
I20260806 04:30:18.928838 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:30:18.939092 9170 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:30:18.940106 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:30:18.947373 9170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f5ebef94684534b2ded4febaf9fdf0: Generated new TSK 0
I20260806 04:30:18.948153 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:30:18.958846 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--builtin_ntp_servers=127.28.94.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 04:30:19.082734 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:19.083053 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:19.083093 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:19.087349 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:19.087430 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:19.087486 9184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:30:19.092833 9184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:19.094116 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:19.095594 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:19.105576 9193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.105612 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.105588 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:19.105914 9184 server_base.cc:1034] running on GCE node
I20260806 04:30:19.106447 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:19.107169 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:19.108332 9184 hybrid_clock.cc:648] HybridClock initialized: now 1785990619108312 us; error 33 us; skew 500 ppm
I20260806 04:30:19.110339 9184 webserver.cc:460] Webserver started at http://127.28.94.1:33457/ using document root <none> and password file <none>
I20260806 04:30:19.110957 9184 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:19.111023 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:19.111248 9184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:19.113595 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "de2d9b1300b74151ba24711c072f86fc"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "06323b21b2dffe65ff882b2cbaa6e7dc"
server_key_iv: "bb012e29015c1b1dd989e033da6f5f7f"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.114185 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "de2d9b1300b74151ba24711c072f86fc"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "06323b21b2dffe65ff882b2cbaa6e7dc"
server_key_iv: "bb012e29015c1b1dd989e033da6f5f7f"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.118261 9184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 04:30:19.121120 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.122408 9184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:30:19.122526 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "de2d9b1300b74151ba24711c072f86fc"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "06323b21b2dffe65ff882b2cbaa6e7dc"
server_key_iv: "bb012e29015c1b1dd989e033da6f5f7f"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.122653 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:19.155453 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:19.156271 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:19.156462 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:19.157160 9184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:19.158222 9184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:19.158274 9184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.158332 9184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:19.158356 9184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.167716 9184 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:40715
I20260806 04:30:19.167750 9341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:40715 every 8 connection(s)
I20260806 04:30:19.168984 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:30:19.177736 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9184
I20260806 04:30:19.177876 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:30:19.178216 29048 external_mini_cluster.cc:1428] Setting key 2c18110b98f5d44fd5a20106908ccdf6
I20260806 04:30:19.179193 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.171059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:43109 (local address 127.28.94.62:38395)
0806 04:30:19.171191 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:19.171195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:19.172443 (+ 1248us) server_negotiation.cc:407] Connection header received
0806 04:30:19.174002 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:19.174006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:19.174073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:19.174147 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:19.175149 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.175695 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.176930 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.177164 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.177258 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.178482 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:19.178486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:19.178489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:19.178506 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:19.178529 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:19.178537 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:19.178602 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:19.178967 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 04:30:19.179043 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:30:19.180255 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.171505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38395 (local address 127.28.94.1:43109)
0806 04:30:19.172270 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:19.172319 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:19.173756 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:19.174366 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:19.174374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:19.174466 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:19.174999 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.175012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.175830 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.175833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.176775 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.176786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.177335 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.177339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.177509 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.178190 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:19.178191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:19.178237 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:19.178319 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:19.178807 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:19.178816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:19.178821 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:30:19.178941 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:30:19.179292 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260806 04:30:19.180358 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--builtin_ntp_servers=127.28.94.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:30:19.181537 9342 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38395
I20260806 04:30:19.181823 9342 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:19.182509 9342 heartbeater.cc:511] Master 127.28.94.62:38395 requested a full tablet report, sending...
I20260806 04:30:19.184324 9110 ts_manager.cc:194] Registered new tserver with Master: de2d9b1300b74151ba24711c072f86fc (127.28.94.1:40715)
I20260806 04:30:19.185984 9110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.1:43109
W20260806 04:30:19.303097 9346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:19.303385 9346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:19.303421 9346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:19.307688 9346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:19.307770 9346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:19.307826 9346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:30:19.312954 9346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:19.314168 9346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:19.315627 9346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:19.326040 9352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.326030 9351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:19.326122 9346 server_base.cc:1034] running on GCE node
W20260806 04:30:19.326038 9355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:19.326869 9346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:19.327646 9346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:19.328817 9346 hybrid_clock.cc:648] HybridClock initialized: now 1785990619328797 us; error 33 us; skew 500 ppm
I20260806 04:30:19.330874 9346 webserver.cc:460] Webserver started at http://127.28.94.2:38915/ using document root <none> and password file <none>
I20260806 04:30:19.331511 9346 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:19.331597 9346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:19.331818 9346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:19.334174 9346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "09b415024a3f45379263b11e672ddecb"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "d52c0ac5ef8c75928c766e8aee5d0e78"
server_key_iv: "2b5d209a8ff61a1506d8d8e76965301d"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.334805 9346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "09b415024a3f45379263b11e672ddecb"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "d52c0ac5ef8c75928c766e8aee5d0e78"
server_key_iv: "2b5d209a8ff61a1506d8d8e76965301d"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.339044 9346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:19.341922 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.343281 9346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:19.343434 9346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "09b415024a3f45379263b11e672ddecb"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "d52c0ac5ef8c75928c766e8aee5d0e78"
server_key_iv: "2b5d209a8ff61a1506d8d8e76965301d"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.343578 9346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:19.369856 9346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:19.370687 9346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:19.370899 9346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:19.371603 9346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:19.372706 9346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:19.372762 9346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.372828 9346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:19.372855 9346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.382764 9346 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42037
I20260806 04:30:19.382787 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42037 every 8 connection(s)
I20260806 04:30:19.383880 9346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:30:19.389247 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9346
I20260806 04:30:19.389371 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:30:19.389671 29048 external_mini_cluster.cc:1428] Setting key ff0620efc5a65fb8a65c44a0c4772452
I20260806 04:30:19.391927 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--builtin_ntp_servers=127.28.94.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:30:19.394761 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.385886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:45009 (local address 127.28.94.62:38395)
0806 04:30:19.386029 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:19.386035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:19.387254 (+ 1219us) server_negotiation.cc:407] Connection header received
0806 04:30:19.388825 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:19.388829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:19.388923 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:19.389025 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:19.390133 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.390770 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.392226 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.392520 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.392651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.393979 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:19.393984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:19.393987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:19.394007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:19.394034 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:19.394047 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:19.394098 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:19.394397 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 04:30:19.394512 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:30:19.395818 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.386321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38395 (local address 127.28.94.2:45009)
0806 04:30:19.387076 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:19.387125 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:19.388571 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:19.389240 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:19.389248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:19.389350 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:19.389965 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.389983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.390926 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.390929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.392018 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.392032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.392690 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.392693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.392863 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.393574 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:19.393576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:19.393627 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:19.393728 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:19.394176 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:19.394189 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:19.394196 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:30:19.394346 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 04:30:19.394720 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260806 04:30:19.397207 9519 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38395
I20260806 04:30:19.397523 9519 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:19.398294 9519 heartbeater.cc:511] Master 127.28.94.62:38395 requested a full tablet report, sending...
I20260806 04:30:19.399680 9110 ts_manager.cc:194] Registered new tserver with Master: 09b415024a3f45379263b11e672ddecb (127.28.94.2:42037)
I20260806 04:30:19.400333 9110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.2:45009
W20260806 04:30:19.518062 9523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:19.518393 9523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:19.518450 9523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:19.522822 9523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:19.522897 9523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:19.522956 9523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:30:19.528136 9523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:19.529433 9523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:19.530977 9523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:19.541404 9537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.541397 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.541407 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:19.541764 9523 server_base.cc:1034] running on GCE node
I20260806 04:30:19.542229 9523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:19.542944 9523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:19.544098 9523 hybrid_clock.cc:648] HybridClock initialized: now 1785990619544068 us; error 45 us; skew 500 ppm
I20260806 04:30:19.546113 9523 webserver.cc:460] Webserver started at http://127.28.94.3:38621/ using document root <none> and password file <none>
I20260806 04:30:19.546727 9523 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:19.546797 9523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:19.547021 9523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:19.549384 9523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "f4877262884a47e188927fca54b86887"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "6f011ef19c754eb056b08142be559f27"
server_key_iv: "4fa90023f5c61554eacfd70749531c97"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.550055 9523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "f4877262884a47e188927fca54b86887"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "6f011ef19c754eb056b08142be559f27"
server_key_iv: "4fa90023f5c61554eacfd70749531c97"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.554252 9523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:30:19.557083 9545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.558393 9523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:30:19.558516 9523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "f4877262884a47e188927fca54b86887"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "6f011ef19c754eb056b08142be559f27"
server_key_iv: "4fa90023f5c61554eacfd70749531c97"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.558655 9523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:19.582253 9523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:19.583042 9523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:19.583236 9523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:19.583892 9523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:19.584986 9523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:19.585039 9523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.585094 9523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:19.585124 9523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.594069 9523 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40369
I20260806 04:30:19.594094 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40369 every 8 connection(s)
I20260806 04:30:19.595086 9523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:30:19.600963 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9523
I20260806 04:30:19.601097 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:30:19.601438 29048 external_mini_cluster.cc:1428] Setting key 452b34dbb65f649a7c9aab68947fb50d
I20260806 04:30:19.605337 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.597138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:38107 (local address 127.28.94.62:38395)
0806 04:30:19.597285 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:19.597289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:19.598519 (+ 1230us) server_negotiation.cc:407] Connection header received
0806 04:30:19.600005 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:19.600008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:19.600074 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:19.600159 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:19.601425 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.602066 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.603436 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.603732 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.603816 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.604886 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:19.604922 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:19.604925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:19.604940 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:19.604965 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:19.604975 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:19.605030 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:19.605161 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 04:30:19.605216 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:30:19.606294 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.597625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38395 (local address 127.28.94.3:38107)
0806 04:30:19.598357 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:19.598401 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:19.599764 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:19.600368 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:19.600377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:19.600473 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:19.601245 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.601269 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.602214 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.602217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.603314 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:19.603326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:19.603873 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:19.603876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:19.604040 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:19.604604 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:19.604605 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:19.604664 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:19.604743 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:19.605031 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:19.605038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:19.605043 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:30:19.605148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:30:19.605414 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260806 04:30:19.607512 9700 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38395
I20260806 04:30:19.607786 9700 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:19.608376 9700 heartbeater.cc:511] Master 127.28.94.62:38395 requested a full tablet report, sending...
I20260806 04:30:19.609543 9110 ts_manager.cc:194] Registered new tserver with Master: f4877262884a47e188927fca54b86887 (127.28.94.3:40369)
I20260806 04:30:19.610154 9110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.3:38107
I20260806 04:30:19.615532 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:30:19.618561 9158 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:19.617676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.28.94.62:38395)
0806 04:30:19.617795 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:19.617799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:19.617939 (+ 140us) server_negotiation.cc:407] Connection header received
0806 04:30:19.618063 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:19.618065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:19.618114 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:19.618181 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:19.618432 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43984: BlockingRecv error: recv got EOF from 127.0.0.1:43984 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:30:19.618721 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9184
I20260806 04:30:19.644304 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9346
I20260806 04:30:19.655102 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9523
I20260806 04:30:19.675486 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9074
2026-08-06T04:30:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1018 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T04:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:19Z Disabled control of system clock
I20260806 04:30:19.717171 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:44815
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:30:19.837239 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:19.837558 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:19.837599 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:19.841820 9713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:19.841902 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:19.841915 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:19.841931 9713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:19.841946 9713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:19.847084 9713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:44815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:19.848317 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:19.849817 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:19.859084 9720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.859069 9722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:19.859069 9724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:19.859419 9713 server_base.cc:1034] running on GCE node
I20260806 04:30:19.860169 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:19.861500 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:19.862684 9713 hybrid_clock.cc:648] HybridClock initialized: now 1785990619862666 us; error 39 us; skew 500 ppm
I20260806 04:30:19.864810 9713 webserver.cc:460] Webserver started at http://127.28.94.62:37615/ using document root <none> and password file <none>
I20260806 04:30:19.865494 9713 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:19.865576 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:19.865820 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:19.868108 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "79c82552cc124e3792fedb1c78175896"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "4a278f8d4102e8fc5049eebd9ff073a8"
server_key_iv: "90b96af857513dbcd1adb499a35deb9a"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.868729 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "79c82552cc124e3792fedb1c78175896"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "4a278f8d4102e8fc5049eebd9ff073a8"
server_key_iv: "90b96af857513dbcd1adb499a35deb9a"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.873027 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:30:19.875778 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:19.877297 9713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:19.877457 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "79c82552cc124e3792fedb1c78175896"
format_stamp: "Formatted at 2026-08-06 04:30:19 on dist-test-slave-ncp9"
server_key: "4a278f8d4102e8fc5049eebd9ff073a8"
server_key_iv: "90b96af857513dbcd1adb499a35deb9a"
server_key_version: "encryptionkey@0"
I20260806 04:30:19.877588 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:19.882870 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:19.883638 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:19.883836 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:19.891341 9713 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:44815
I20260806 04:30:19.891353 9802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:44815 every 8 connection(s)
I20260806 04:30:19.892400 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:30:19.894217 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9713
I20260806 04:30:19.894344 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:30:19.894666 29048 external_mini_cluster.cc:1428] Setting key 600da5a76b28c2d67a63c497b5da5982
I20260806 04:30:19.896581 9803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:19.900560 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:19.896170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.28.94.62:44815)
0806 04:30:19.896877 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:19.896919 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:19.896967 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:30:19.897647 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:19.897666 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:19.898038 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:19.898335 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:19.899174 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:19.899179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:19.899192 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:19.899203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:19.899263 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:19.899281 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:19.899399 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:19.899518 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 04:30:19.899788 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260806 04:30:19.903643 9803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896: Bootstrap starting.
I20260806 04:30:19.906117 9803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:19.906940 9803 log.cc:824] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:19.909377 9803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896: No bootstrap required, opened a new log
I20260806 04:30:19.912923 9803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } }
I20260806 04:30:19.913239 9803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:19.913335 9803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c82552cc124e3792fedb1c78175896, State: Initialized, Role: FOLLOWER
I20260806 04:30:19.913821 9803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } }
I20260806 04:30:19.913970 9803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:30:19.914040 9803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:30:19.914153 9803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:19.915202 9803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } }
I20260806 04:30:19.915575 9803 leader_election.cc:302] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c82552cc124e3792fedb1c78175896; no voters:
I20260806 04:30:19.915926 9803 leader_election.cc:288] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:30:19.916054 9808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:30:19.916301 9808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [term 1 LEADER]: Becoming Leader. State: Replica: 79c82552cc124e3792fedb1c78175896, State: Running, Role: LEADER
I20260806 04:30:19.916710 9808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } }
I20260806 04:30:19.917433 9803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:30:19.918705 9810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c82552cc124e3792fedb1c78175896. Latest consensus state: current_term: 1 leader_uuid: "79c82552cc124e3792fedb1c78175896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } } }
I20260806 04:30:19.918705 9809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c82552cc124e3792fedb1c78175896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c82552cc124e3792fedb1c78175896" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 44815 } } }
I20260806 04:30:19.918862 9810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:19.918880 9809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:19.919317 9817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:30:19.922384 9817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:30:19.928681 9817 catalog_manager.cc:1347] Generated new cluster ID: ed12317c04ab45bf9ab04c73a8f23356
I20260806 04:30:19.928777 9817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:30:19.937489 9817 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:30:19.937848 9817 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:30:19.948849 9817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c82552cc124e3792fedb1c78175896: Generated new TSK 0
I20260806 04:30:19.949668 9817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:30:19.960404 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--builtin_ntp_servers=127.28.94.20:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:30:20.083822 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:20.084141 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:20.084189 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:20.088481 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:20.088573 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:20.088650 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:30:20.093858 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:20.095124 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:20.096649 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:20.106966 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.106998 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.106958 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:20.107426 9832 server_base.cc:1034] running on GCE node
I20260806 04:30:20.107941 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:20.108660 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:20.109838 9832 hybrid_clock.cc:648] HybridClock initialized: now 1785990620109794 us; error 55 us; skew 500 ppm
I20260806 04:30:20.111886 9832 webserver.cc:460] Webserver started at http://127.28.94.1:40453/ using document root <none> and password file <none>
I20260806 04:30:20.112527 9832 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:20.112603 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:20.112820 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:20.115099 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "326e6da428e5481b9347366250f6f18a"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "fafdaa70174d1b4722856cffe1cd7292"
server_key_iv: "2aa18d9575f6fc0589d3795a42937812"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.115711 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "326e6da428e5481b9347366250f6f18a"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "fafdaa70174d1b4722856cffe1cd7292"
server_key_iv: "2aa18d9575f6fc0589d3795a42937812"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.120090 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 04:30:20.123134 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.124508 9832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:30:20.124675 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "326e6da428e5481b9347366250f6f18a"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "fafdaa70174d1b4722856cffe1cd7292"
server_key_iv: "2aa18d9575f6fc0589d3795a42937812"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.124799 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:20.129660 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:20.130357 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:20.130534 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:20.131143 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:20.132153 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:20.132205 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.132258 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:20.132287 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.141273 9832 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:34105
I20260806 04:30:20.141314 10001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:34105 every 8 connection(s)
I20260806 04:30:20.142319 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:30:20.148592 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.144291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:46523 (local address 127.28.94.62:44815)
0806 04:30:20.144445 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:20.144449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:20.145622 (+ 1173us) server_negotiation.cc:407] Connection header received
0806 04:30:20.146961 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:20.146964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:20.147006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:20.147081 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:20.148181 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:20.148183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:20.148185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:20.148194 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:20.148209 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:20.148216 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:20.148287 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:20.148374 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 04:30:20.148417 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:30:20.148749 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 9832
I20260806 04:30:20.148857 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:30:20.149195 29048 external_mini_cluster.cc:1428] Setting key d0d7805a3d67316d08af46d5cbe758b8
I20260806 04:30:20.149602 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.144732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:44815 (local address 127.28.94.1:46523)
0806 04:30:20.145472 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:20.145518 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:20.146759 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:20.147233 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:20.147240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:20.147338 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:20.147914 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:20.147915 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:20.147958 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:20.148048 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:20.148267 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:20.148271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:20.148275 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:30:20.148406 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 04:30:20.148641 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":150,"threads_started":1}
I20260806 04:30:20.150880 10003 heartbeater.cc:347] Connected to a master server at 127.28.94.62:44815
I20260806 04:30:20.151139 10003 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:20.151391 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--builtin_ntp_servers=127.28.94.20:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:30:20.151645 10003 heartbeater.cc:511] Master 127.28.94.62:44815 requested a full tablet report, sending...
I20260806 04:30:20.153486 9754 ts_manager.cc:194] Registered new tserver with Master: 326e6da428e5481b9347366250f6f18a (127.28.94.1:34105)
W20260806 04:30:20.276711 10006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:20.277045 10006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:20.277094 10006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:20.281275 10006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:20.281345 10006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:20.281409 10006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:30:20.286506 10006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:20.287751 10006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:20.289290 10006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:20.300365 10012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.300365 10011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.300369 10014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:20.300669 10006 server_base.cc:1034] running on GCE node
I20260806 04:30:20.301206 10006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:20.301944 10006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:20.303131 10006 hybrid_clock.cc:648] HybridClock initialized: now 1785990620303103 us; error 38 us; skew 500 ppm
I20260806 04:30:20.305193 10006 webserver.cc:460] Webserver started at http://127.28.94.2:41391/ using document root <none> and password file <none>
I20260806 04:30:20.305823 10006 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:20.305886 10006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:20.306075 10006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:20.308284 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "34695adc22a14e6da2d079c1bcc7245e"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "82022132bb505cf713a4bf04472aee3d"
server_key_iv: "fac75d36390205c8b006bb62e26bc4cb"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.308862 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "34695adc22a14e6da2d079c1bcc7245e"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "82022132bb505cf713a4bf04472aee3d"
server_key_iv: "fac75d36390205c8b006bb62e26bc4cb"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.313011 10006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:30:20.315797 10022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.317191 10006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:20.317344 10006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "34695adc22a14e6da2d079c1bcc7245e"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "82022132bb505cf713a4bf04472aee3d"
server_key_iv: "fac75d36390205c8b006bb62e26bc4cb"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.317467 10006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:20.322381 10006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:20.323055 10006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:20.323227 10006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:20.323838 10006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:20.324821 10006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:20.324872 10006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.324970 10006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:20.324997 10006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.334112 10006 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:34931
I20260806 04:30:20.334156 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:34931 every 8 connection(s)
I20260806 04:30:20.335127 10006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:30:20.339619 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10006
I20260806 04:30:20.339728 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:30:20.340034 29048 external_mini_cluster.cc:1428] Setting key a8280b18917a76dd398e952e6d00c417
I20260806 04:30:20.341835 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.337051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46305 (local address 127.28.94.62:44815)
0806 04:30:20.337189 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:20.337193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:20.338429 (+ 1236us) server_negotiation.cc:407] Connection header received
0806 04:30:20.339839 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:20.339843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:20.339897 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:20.339990 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:20.341334 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:20.341337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:20.341340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:20.341353 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:20.341376 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:20.341385 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:20.341464 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:20.341656 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 04:30:20.341700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:30:20.342382 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--builtin_ntp_servers=127.28.94.20:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:30:20.342897 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.337485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:44815 (local address 127.28.94.2:46305)
0806 04:30:20.338274 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:20.338317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:20.339586 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:20.340191 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:20.340199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:20.340296 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:20.341024 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:20.341026 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:20.341085 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:20.341182 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:20.341468 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:20.341478 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:20.341485 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:30:20.341707 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 04:30:20.341980 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":136,"threads_started":1}
I20260806 04:30:20.344218 10176 heartbeater.cc:347] Connected to a master server at 127.28.94.62:44815
I20260806 04:30:20.344501 10176 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:20.345065 10176 heartbeater.cc:511] Master 127.28.94.62:44815 requested a full tablet report, sending...
I20260806 04:30:20.346354 9754 ts_manager.cc:194] Registered new tserver with Master: 34695adc22a14e6da2d079c1bcc7245e (127.28.94.2:34931)
W20260806 04:30:20.469489 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:20.469796 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:20.469838 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:20.474030 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:20.474114 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:20.474175 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:30:20.479195 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:20.480451 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:20.482028 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:20.492496 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.492508 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:20.492527 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:20.494470 10184 server_base.cc:1034] running on GCE node
I20260806 04:30:20.494987 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:20.495751 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:20.496968 10184 hybrid_clock.cc:648] HybridClock initialized: now 1785990620496941 us; error 38 us; skew 500 ppm
I20260806 04:30:20.498936 10184 webserver.cc:460] Webserver started at http://127.28.94.3:45955/ using document root <none> and password file <none>
I20260806 04:30:20.499562 10184 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:20.499636 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:20.499854 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:20.502228 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "d46d75d934c043b383c122ab2be6583c"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "5d07af0fbd5a54c53b7642dce7b50f7d"
server_key_iv: "c082619cdabcb85c613dc6f5d891b486"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.502918 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "d46d75d934c043b383c122ab2be6583c"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "5d07af0fbd5a54c53b7642dce7b50f7d"
server_key_iv: "c082619cdabcb85c613dc6f5d891b486"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.507170 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:20.510110 10200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.511556 10184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:20.511703 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "d46d75d934c043b383c122ab2be6583c"
format_stamp: "Formatted at 2026-08-06 04:30:20 on dist-test-slave-ncp9"
server_key: "5d07af0fbd5a54c53b7642dce7b50f7d"
server_key_iv: "c082619cdabcb85c613dc6f5d891b486"
server_key_version: "encryptionkey@0"
I20260806 04:30:20.511833 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:20.516917 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:20.517670 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:20.517922 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:20.518604 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:20.519723 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:20.519773 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.519829 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:20.519856 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:20.529217 10184 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:35487
I20260806 04:30:20.529239 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:35487 every 8 connection(s)
I20260806 04:30:20.530280 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:30:20.530942 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10184
I20260806 04:30:20.531054 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:30:20.531353 29048 external_mini_cluster.cc:1428] Setting key 772d852597707eef115c68f6cd9f2557
I20260806 04:30:20.537010 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.532354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:49851 (local address 127.28.94.62:44815)
0806 04:30:20.532537 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:20.532542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:20.533921 (+ 1379us) server_negotiation.cc:407] Connection header received
0806 04:30:20.535269 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:20.535272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:20.535320 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:20.535396 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:20.536511 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:20.536515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:20.536517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:30:20.536530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:20.536551 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:30:20.536559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:20.536647 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:20.536783 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 04:30:20.536834 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 04:30:20.537955 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:20.532927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:44815 (local address 127.28.94.3:49851)
0806 04:30:20.533774 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:20.533820 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:20.535042 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:20.535601 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:20.535609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:20.535706 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:20.536246 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:30:20.536247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:20.536291 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:30:20.536361 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:20.536630 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:20.536640 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:20.536646 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:30:20.536822 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 04:30:20.537135 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":123,"threads_started":1}
I20260806 04:30:20.539101 10353 heartbeater.cc:347] Connected to a master server at 127.28.94.62:44815
I20260806 04:30:20.539350 10353 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:20.539803 10353 heartbeater.cc:511] Master 127.28.94.62:44815 requested a full tablet report, sending...
I20260806 04:30:20.540884 9754 ts_manager.cc:194] Registered new tserver with Master: d46d75d934c043b383c122ab2be6583c (127.28.94.3:35487)
I20260806 04:30:20.545588 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:30:20.548475 9806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:30:20.547634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.28.94.62:44815)
0806 04:30:20.547798 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:20.547802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:20.547941 (+ 139us) server_negotiation.cc:407] Connection header received
0806 04:30:20.548059 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:20.548061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:20.548097 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:20.548158 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:20.548352 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:30:20.548638 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9832
I20260806 04:30:20.575069 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10006
I20260806 04:30:20.594302 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10184
I20260806 04:30:20.607350 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 9713
2026-08-06T04:30:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (928 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:30:20 dist-test-slave-ncp9 krb5kdc[10365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:30:20 dist-test-slave-ncp9 krb5kdc[10365](info): set up 2 sockets
Aug 06 04:30:20 dist-test-slave-ncp9 krb5kdc[10365](info): commencing operation
krb5kdc: starting...
W20260806 04:30:21.726001 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:30:21 dist-test-slave-ncp9 krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:30:21.858144 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38043
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:38043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:21.979272 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:21.979593 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:21.979655 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:21.983866 10391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:30:21.983952 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:21.983979 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:21.984000 10391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:30:21.984021 10391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:30:21.989763 10391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:38043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:21.991003 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:21.992398 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:22.001719 10397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.001718 10401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.001719 10398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:22.002123 10391 server_base.cc:1034] running on GCE node
I20260806 04:30:22.002775 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:22.004019 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:22.005180 10391 hybrid_clock.cc:648] HybridClock initialized: now 1785990622005160 us; error 36 us; skew 500 ppm
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:22.008060 10391 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:30:22.009125 10391 webserver.cc:460] Webserver started at http://127.28.94.62:34435/ using document root <none> and password file <none>
I20260806 04:30:22.009711 10391 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:22.009773 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:22.009964 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:22.012199 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "54db37d955f54c1aa9ba466cee40f557"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "6b88d67557382d9e0970911a26bd73b6"
server_key_iv: "0745cb6fcf45741ca1331479c8a0cd6f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.012775 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "54db37d955f54c1aa9ba466cee40f557"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "6b88d67557382d9e0970911a26bd73b6"
server_key_iv: "0745cb6fcf45741ca1331479c8a0cd6f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.017005 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:30:22.019719 10412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.021178 10391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:30:22.021325 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "54db37d955f54c1aa9ba466cee40f557"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "6b88d67557382d9e0970911a26bd73b6"
server_key_iv: "0745cb6fcf45741ca1331479c8a0cd6f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.021446 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:22.034446 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:22.037993 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:22.038221 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:22.045760 10391 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:38043
I20260806 04:30:22.046051 10475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:38043 every 8 connection(s)
I20260806 04:30:22.046887 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:30:22.050007 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:22.056214 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: Bootstrap starting.
I20260806 04:30:22.056551 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10391
I20260806 04:30:22.056665 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:30:22.056984 29048 external_mini_cluster.cc:1428] Setting key 41a2fc5f7d1207b4235abb300c97599c
I20260806 04:30:22.059015 10476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:22.060009 10476 log.cc:824] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:22.062624 10476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: No bootstrap required, opened a new log
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:22.066874 10476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } }
I20260806 04:30:22.067209 10476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:22.067277 10476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54db37d955f54c1aa9ba466cee40f557, State: Initialized, Role: FOLLOWER
I20260806 04:30:22.067801 10476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } }
I20260806 04:30:22.067945 10476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:30:22.068018 10476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:30:22.068113 10476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:22.069130 10476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } }
I20260806 04:30:22.069495 10476 leader_election.cc:302] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54db37d955f54c1aa9ba466cee40f557; no voters:
I20260806 04:30:22.069880 10476 leader_election.cc:288] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:30:22.069995 10482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:30:22.070243 10482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [term 1 LEADER]: Becoming Leader. State: Replica: 54db37d955f54c1aa9ba466cee40f557, State: Running, Role: LEADER
I20260806 04:30:22.070660 10482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } }
I20260806 04:30:22.071453 10476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:30:22.072633 10484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54db37d955f54c1aa9ba466cee40f557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } } }
I20260806 04:30:22.072765 10484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:22.073256 10485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54db37d955f54c1aa9ba466cee40f557. Latest consensus state: current_term: 1 leader_uuid: "54db37d955f54c1aa9ba466cee40f557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db37d955f54c1aa9ba466cee40f557" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38043 } } }
I20260806 04:30:22.073401 10485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557 [sys.catalog]: This master's current role is: LEADER
I20260806 04:30:22.073674 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.058507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.28.94.62:38043)
0806 04:30:22.059018 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.059040 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.059090 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:30:22.060020 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.060049 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.060524 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.060840 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:22.061645 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.063137 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.064034 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.064786 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.064992 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.067669 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:22.067675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:22.067691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:22.067727 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:22.069497 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.070063 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.070070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.070077 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.070244 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.070589 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.070592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.070593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.071157 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:22.071370 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.072145 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.072502 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":130,"threads_started":1}
W20260806 04:30:22.077037 10506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:30:22.077131 10506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:30:22.077209 10494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:30:22.078187 10494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:30:22.084692 10494 catalog_manager.cc:1347] Generated new cluster ID: 17fee321c7a94610839fce58ce431da1
I20260806 04:30:22.084775 10494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:30:22.100375 10494 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:30:22.101842 10494 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:30:22.110404 10494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: Generated new TSK 0
I20260806 04:30:22.111382 10494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:30:22.150574 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:22.272523 10512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:22.272854 10512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:22.272949 10512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:22.277163 10512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:22.277252 10512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:22.277338 10512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:30:22.283041 10512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:22.284292 10512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:22.285459 10512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:22.295382 10521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.295356 10523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.295419 10520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:22.295714 10512 server_base.cc:1034] running on GCE node
I20260806 04:30:22.296237 10512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:22.296980 10512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:22.298180 10512 hybrid_clock.cc:648] HybridClock initialized: now 1785990622298159 us; error 35 us; skew 500 ppm
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:22.301856 10512 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:30:22.302876 10512 webserver.cc:460] Webserver started at http://127.28.94.1:39963/ using document root <none> and password file <none>
I20260806 04:30:22.303449 10512 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:22.303509 10512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:22.303699 10512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:22.305892 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "bbd4229fd21f4fe1a8700322ad3587a8"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "63b95c8a4a4b16ec3fd6d5ff739e0ede"
server_key_iv: "2a5782eff5192893c4e99edd8bc8a560"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.306454 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "bbd4229fd21f4fe1a8700322ad3587a8"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "63b95c8a4a4b16ec3fd6d5ff739e0ede"
server_key_iv: "2a5782eff5192893c4e99edd8bc8a560"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.310539 10512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:22.313187 10532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.314482 10512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:30:22.314604 10512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "bbd4229fd21f4fe1a8700322ad3587a8"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "63b95c8a4a4b16ec3fd6d5ff739e0ede"
server_key_iv: "2a5782eff5192893c4e99edd8bc8a560"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.314707 10512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:22.359223 10512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:22.362347 10512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:22.362526 10512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:22.363119 10512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:22.364001 10512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:22.364050 10512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.364094 10512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:22.364113 10512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.373319 10512 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:38629
I20260806 04:30:22.373343 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:38629 every 8 connection(s)
I20260806 04:30:22.374330 10512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:30:22.379696 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10512
I20260806 04:30:22.379845 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:30:22.380160 29048 external_mini_cluster.cc:1428] Setting key 499376a060613cc615fcffd559b424f4
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:22.389179 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.376280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:52843 (local address 127.28.94.62:38043)
0806 04:30:22.376424 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.376428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.377396 (+ 968us) server_negotiation.cc:407] Connection header received
0806 04:30:22.378776 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.378779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.378835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.378909 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:22.380410 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.380990 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.382539 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.382778 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.382861 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.385636 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:22.385640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:22.385642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:22.385676 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:22.387007 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.387549 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.387553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.387555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.387638 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.387982 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.387985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.387987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.388191 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:22.388279 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.388832 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.389002 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:30:22.390105 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.376704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38043 (local address 127.28.94.1:52843)
0806 04:30:22.377221 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.377269 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.378564 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.379054 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.379060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.379485 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:22.380239 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.380264 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.381125 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.381129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.382413 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.382425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.382943 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.382947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.383149 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.383801 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:22.383826 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:22.385441 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:22.387130 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.387137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.387167 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.387445 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.387731 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.387735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.387737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.387890 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.388292 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:22.388300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:22.388584 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.388773 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.389201 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:30:22.391443 10694 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38043
I20260806 04:30:22.391709 10694 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:22.392261 10694 heartbeater.cc:511] Master 127.28.94.62:38043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:30:22.393975 10432 ts_manager.cc:194] Registered new tserver with Master: bbd4229fd21f4fe1a8700322ad3587a8 (127.28.94.1:38629)
I20260806 04:30:22.395125 10432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:52843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:30:22.407999 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:22.530392 10699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:22.530714 10699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:22.530763 10699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:22.535070 10699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:22.535166 10699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:22.535259 10699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:30:22.541095 10699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:22.542275 10699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:22.543345 10699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:22.553361 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.553363 10707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.553361 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:22.553695 10699 server_base.cc:1034] running on GCE node
I20260806 04:30:22.554255 10699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:22.554947 10699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:22.556123 10699 hybrid_clock.cc:648] HybridClock initialized: now 1785990622556100 us; error 39 us; skew 500 ppm
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:22.559044 10699 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:30:22.560022 10699 webserver.cc:460] Webserver started at http://127.28.94.2:38361/ using document root <none> and password file <none>
I20260806 04:30:22.560645 10699 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:22.560719 10699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:22.561002 10699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:22.563220 10699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "4a53874d1fd84f1b91162b6a76521cdd"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "9593b73c2d4f01af687ae79d69ffbc2f"
server_key_iv: "63b14802c97641042f849c81d6b7794f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.563835 10699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "4a53874d1fd84f1b91162b6a76521cdd"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "9593b73c2d4f01af687ae79d69ffbc2f"
server_key_iv: "63b14802c97641042f849c81d6b7794f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.568030 10699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:30:22.570804 10722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.572258 10699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:30:22.572394 10699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "4a53874d1fd84f1b91162b6a76521cdd"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "9593b73c2d4f01af687ae79d69ffbc2f"
server_key_iv: "63b14802c97641042f849c81d6b7794f"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.572515 10699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:22.592782 10699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:22.596030 10699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:22.596240 10699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:22.596936 10699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:22.597863 10699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:22.597913 10699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.597970 10699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:22.597994 10699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.607029 10699 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:44481
I20260806 04:30:22.607053 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:44481 every 8 connection(s)
I20260806 04:30:22.607995 10699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:30:22.616828 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10699
I20260806 04:30:22.616984 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:30:22.617281 29048 external_mini_cluster.cc:1428] Setting key bfb99d1607652b854250cdb743d59605
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:22.622875 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.609960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46447 (local address 127.28.94.62:38043)
0806 04:30:22.610128 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.610133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.611364 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 04:30:22.612931 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.612935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.613000 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.613094 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:22.614422 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.614969 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.616147 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.616373 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.616463 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.619364 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:22.619367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:22.619369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:22.619394 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:22.620657 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.621315 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.621319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.621320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.621416 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.621737 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.621740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.621741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.621970 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:22.622065 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.622584 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.622719 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:30:22.623736 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.610552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38043 (local address 127.28.94.2:46447)
0806 04:30:22.611197 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.611251 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.612672 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.613304 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.613311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.613727 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:22.614260 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.614275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.615105 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.615108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.616020 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.616030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.616532 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.616536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.616727 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.617373 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:22.617396 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:22.619164 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:22.620781 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.620789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.620808 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.621216 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.621503 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.621505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.621507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.621658 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.622061 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:22.622066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:22.622346 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.622507 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.622789 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":145,"threads_started":1}
I20260806 04:30:22.625029 10877 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38043
I20260806 04:30:22.625303 10877 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:22.625883 10877 heartbeater.cc:511] Master 127.28.94.62:38043 requested a full tablet report, sending...
I20260806 04:30:22.627111 10432 ts_manager.cc:194] Registered new tserver with Master: 4a53874d1fd84f1b91162b6a76521cdd (127.28.94.2:44481)
I20260806 04:30:22.627712 10432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:46447
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:30:22.645430 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:30:22.766026 10884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:30:22.766335 10884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:30:22.766391 10884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:30:22.770545 10884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:30:22.770625 10884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:30:22.770705 10884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:30:22.776392 10884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:30:22.777644 10884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:30:22.778687 10884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:30:22.788825 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:22.788885 10884 server_base.cc:1034] running on GCE node
W20260806 04:30:22.788831 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:30:22.788829 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:30:22.789674 10884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:30:22.790345 10884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:30:22.791510 10884 hybrid_clock.cc:648] HybridClock initialized: now 1785990622791479 us; error 46 us; skew 500 ppm
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:30:22.794441 10884 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:30:22.795461 10884 webserver.cc:460] Webserver started at http://127.28.94.3:36363/ using document root <none> and password file <none>
I20260806 04:30:22.796069 10884 fs_manager.cc:359] Metadata directory not provided
I20260806 04:30:22.796139 10884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:30:22.796362 10884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:30:22.798648 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "396e6d276d8042e6b1bf397922a18574"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "bf7693f29e453514a5fd84f803b2df00"
server_key_iv: "7b99907187c2ee05c474c57688f83320"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.799248 10884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "396e6d276d8042e6b1bf397922a18574"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "bf7693f29e453514a5fd84f803b2df00"
server_key_iv: "7b99907187c2ee05c474c57688f83320"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.803290 10884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:30:22.805948 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.807236 10884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:30:22.807363 10884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "396e6d276d8042e6b1bf397922a18574"
format_stamp: "Formatted at 2026-08-06 04:30:22 on dist-test-slave-ncp9"
server_key: "bf7693f29e453514a5fd84f803b2df00"
server_key_iv: "7b99907187c2ee05c474c57688f83320"
server_key_version: "encryptionkey@0"
I20260806 04:30:22.807476 10884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:30:22.816823 10884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:30:22.820262 10884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:30:22.820465 10884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:30:22.821151 10884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:30:22.822069 10884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:30:22.822121 10884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.822180 10884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:30:22.822222 10884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:30:22.832050 10884 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40159
I20260806 04:30:22.832069 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40159 every 8 connection(s)
I20260806 04:30:22.833178 10884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:30:22.843406 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 10884
Aug 06 04:30:22 dist-test-slave-ncp9 krb5kdc[10365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990622, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:30:22.843560 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:30:22.843849 29048 external_mini_cluster.cc:1428] Setting key 955cb9d8b46f1f3e8fd7aed22998f52a
I20260806 04:30:22.847954 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.835259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:33533 (local address 127.28.94.62:38043)
0806 04:30:22.835455 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.835459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.836368 (+ 909us) server_negotiation.cc:407] Connection header received
0806 04:30:22.837942 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.837946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.838023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.838099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:22.839424 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.839960 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.841224 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.841445 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.841543 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.844223 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:22.844226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:22.844228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:22.844257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:22.845781 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.846346 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.846350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.846352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.846436 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.846778 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.846782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.846784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.846990 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:22.847082 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.847653 (+ 571us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.847786 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 04:30:22.848862 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.835647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38043 (local address 127.28.94.3:33533)
0806 04:30:22.836191 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.836241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.837694 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.838311 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.838318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.838729 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:30:22.839263 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.839279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.840102 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.840105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.841096 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.841109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.841616 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.841620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.841796 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.842420 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:30:22.842440 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:30:22.844018 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:30:22.845910 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.845917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.845937 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.846241 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.846523 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:30:22.846526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:30:22.846529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:30:22.846686 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:30:22.847103 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:30:22.847111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:30:22.847394 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.847582 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.847964 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":137,"threads_started":1}
I20260806 04:30:22.850073 11086 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38043
I20260806 04:30:22.850323 11086 heartbeater.cc:464] Registering TS with master...
I20260806 04:30:22.850865 11086 heartbeater.cc:511] Master 127.28.94.62:38043 requested a full tablet report, sending...
I20260806 04:30:22.852025 10432 ts_manager.cc:194] Registered new tserver with Master: 396e6d276d8042e6b1bf397922a18574 (127.28.94.3:40159)
I20260806 04:30:22.852622 10432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:33533
I20260806 04:30:22.858007 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:30:22.867003 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.859995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.28.94.62:38043)
0806 04:30:22.860126 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.860129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.860302 (+ 173us) server_negotiation.cc:407] Connection header received
0806 04:30:22.860405 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.860407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.860449 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.860523 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:30:22.861286 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.861816 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.862623 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.862828 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.862898 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.864274 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:30:22.864277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:30:22.864279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:30:22.864300 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:30:22.865507 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.865902 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.865905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.865906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.865988 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:30:22.866276 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:30:22.866278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:30:22.866279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:30:22.866500 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:30:22.866604 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.866777 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.866875 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:30:22.870071 10432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:30:22.873437 10432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:30:22.889192 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.882634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.28.94.1:38629)
0806 04:30:22.883634 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.883653 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.883679 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:30:22.883760 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.883765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.883996 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.884170 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:30:22.884545 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.885809 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.887448 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.888336 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.888454 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.888622 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.888869 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.889013 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":90,"threads_started":1}
I20260806 04:30:22.889194 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.882623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:44481 (local address 127.0.0.1:60322)
0806 04:30:22.883496 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.883509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.883587 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.884304 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.884307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.884321 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:30:22.884488 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.884498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.886081 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.886083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.887441 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.887454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.888483 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.888488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.888627 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.888693 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.888820 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.889015 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":241,"thread_start_us":119,"threads_started":1}
I20260806 04:30:22.889195 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.882465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:38629 (local address 127.0.0.1:33844)
0806 04:30:22.883029 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.883081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.883215 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.884167 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.884174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.884197 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:30:22.884409 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.884418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.885939 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.885942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.887305 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.887317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.888483 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.888488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.888661 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.888703 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.888820 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.889011 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":117,"spinlock_wait_cycles":1920,"thread_start_us":99,"threads_started":1}
I20260806 04:30:22.890029 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.883294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.28.94.2:44481)
0806 04:30:22.883843 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.883855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.883877 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:30:22.883959 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.883964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.884155 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.884439 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:30:22.884604 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.885809 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.887581 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.888353 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.888454 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.888622 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.889802 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.889902 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":121,"threads_started":1}
I20260806 04:30:22.891319 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.883215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40159 (local address 127.0.0.1:45928)
0806 04:30:22.884159 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.884171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.884266 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.884742 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.884744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.884763 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:30:22.884853 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.884859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.885819 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.885822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.887441 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.887454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.890704 (+ 3250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.890707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.890995 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.891023 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.891102 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.891180 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":75,"spinlock_wait_cycles":1408,"thread_start_us":94,"threads_started":1}
I20260806 04:30:22.891834 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.883530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.28.94.3:40159)
0806 04:30:22.883980 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.883991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.884356 (+ 365us) server_negotiation.cc:407] Connection header received
0806 04:30:22.884433 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.884438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.884625 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.884751 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:30:22.884962 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.885696 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.888691 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.889537 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.889650 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.889797 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.891557 (+ 1760us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.891676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":150,"threads_started":1}
I20260806 04:30:22.892834 10593 tablet_service.cc:1467] Processing CreateTablet for tablet 9fffa47fd545432593f92076f534b733 (DEFAULT_TABLE table=test-table [id=b500e03e878e4767b5ad2fd3480445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:30:22.892951 10796 tablet_service.cc:1467] Processing CreateTablet for tablet 9fffa47fd545432593f92076f534b733 (DEFAULT_TABLE table=test-table [id=b500e03e878e4767b5ad2fd3480445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:30:22.893954 10593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fffa47fd545432593f92076f534b733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:22.894035 10796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fffa47fd545432593f92076f534b733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:22.894611 10977 tablet_service.cc:1467] Processing CreateTablet for tablet 9fffa47fd545432593f92076f534b733 (DEFAULT_TABLE table=test-table [id=b500e03e878e4767b5ad2fd3480445b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:30:22.895689 10977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fffa47fd545432593f92076f534b733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:30:22.900812 11112 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: Bootstrap starting.
I20260806 04:30:22.902350 11113 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Bootstrap starting.
I20260806 04:30:22.903223 11112 tablet_bootstrap.cc:654] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:22.903772 11114 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Bootstrap starting.
I20260806 04:30:22.904161 11112 log.cc:824] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:22.905109 11113 tablet_bootstrap.cc:654] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:22.905946 11113 log.cc:824] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:22.906258 11114 tablet_bootstrap.cc:654] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Neither blocks nor log segments found. Creating new log.
I20260806 04:30:22.907109 11112 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: No bootstrap required, opened a new log
I20260806 04:30:22.907335 11112 ts_tablet_manager.cc:1397] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 04:30:22.907603 11114 log.cc:824] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Log is configured to *not* fsync() on all Append() calls
I20260806 04:30:22.907732 11113 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: No bootstrap required, opened a new log
I20260806 04:30:22.907915 11113 ts_tablet_manager.cc:1397] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:30:22.909817 11114 tablet_bootstrap.cc:492] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: No bootstrap required, opened a new log
I20260806 04:30:22.910039 11114 ts_tablet_manager.cc:1397] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:30:22.911044 11113 raft_consensus.cc:348] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.911342 11113 raft_consensus.cc:374] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:22.911401 11113 raft_consensus.cc:729] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a53874d1fd84f1b91162b6a76521cdd, State: Initialized, Role: FOLLOWER
I20260806 04:30:22.911840 11113 consensus_queue.cc:260] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.911815 11112 raft_consensus.cc:348] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.912737 10877 heartbeater.cc:503] Master 127.28.94.62:38043 was elected leader, sending a full tablet report...
I20260806 04:30:22.912969 11113 ts_tablet_manager.cc:1428] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:30:22.912101 11112 raft_consensus.cc:374] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:22.913883 11112 raft_consensus.cc:729] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd4229fd21f4fe1a8700322ad3587a8, State: Initialized, Role: FOLLOWER
I20260806 04:30:22.913957 11114 raft_consensus.cc:348] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.914228 11114 raft_consensus.cc:374] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:30:22.914286 11114 raft_consensus.cc:729] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396e6d276d8042e6b1bf397922a18574, State: Initialized, Role: FOLLOWER
I20260806 04:30:22.914397 11112 consensus_queue.cc:260] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.914686 11114 consensus_queue.cc:260] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.915215 10694 heartbeater.cc:503] Master 127.28.94.62:38043 was elected leader, sending a full tablet report...
I20260806 04:30:22.915539 11114 ts_tablet_manager.cc:1428] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:30:22.915544 11112 ts_tablet_manager.cc:1428] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20260806 04:30:22.915886 11086 heartbeater.cc:503] Master 127.28.94.62:38043 was elected leader, sending a full tablet report...
I20260806 04:30:22.932543 11119 raft_consensus.cc:482] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:30:22.932808 11119 raft_consensus.cc:504] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.934060 11119 leader_election.cc:288] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd4229fd21f4fe1a8700322ad3587a8 (127.28.94.1:38629), 396e6d276d8042e6b1bf397922a18574 (127.28.94.3:40159)
I20260806 04:30:22.937740 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.934333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:38629 (local address 127.28.94.2:44079)
0806 04:30:22.934472 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.934489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.934604 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.934867 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.934870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.934892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:30:22.935011 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.935017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.935744 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.935747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.936609 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.936619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.937318 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.937322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.937425 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.937448 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.937514 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.937593 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":20}
I20260806 04:30:22.937739 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.934451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:44079 (local address 127.28.94.1:38629)
0806 04:30:22.934569 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.934575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.934601 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:30:22.934704 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.934707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.934755 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.934832 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:30:22.935124 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.935633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.936734 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.937200 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.937298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.937368 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.937544 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.937628 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:30:22.938271 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.934638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40159 (local address 127.28.94.2:58199)
0806 04:30:22.935093 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:22.935107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:30:22.935206 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:30:22.935421 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:30:22.935424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:30:22.935441 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:30:22.935560 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.935566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.936284 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.936288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.937159 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:30:22.937169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.937819 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.937823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:30:22.937916 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.937942 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:30:22.938059 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:30:22.938131 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
I20260806 04:30:22.938298 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.934792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:58199 (local address 127.28.94.3:40159)
0806 04:30:22.934920 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.934925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.935192 (+ 267us) server_negotiation.cc:407] Connection header received
0806 04:30:22.935293 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.935296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.935339 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.935399 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:30:22.935665 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.936182 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.937278 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.937687 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.937783 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.937844 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.938112 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.938201 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:30:22.938547 10624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fffa47fd545432593f92076f534b733" candidate_uuid: "4a53874d1fd84f1b91162b6a76521cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" is_pre_election: true
I20260806 04:30:22.938905 10624 raft_consensus.cc:2454] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a53874d1fd84f1b91162b6a76521cdd in term 0.
I20260806 04:30:22.938958 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fffa47fd545432593f92076f534b733" candidate_uuid: "4a53874d1fd84f1b91162b6a76521cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396e6d276d8042e6b1bf397922a18574" is_pre_election: true
I20260806 04:30:22.939216 11003 raft_consensus.cc:2454] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a53874d1fd84f1b91162b6a76521cdd in term 0.
I20260806 04:30:22.939406 10723 leader_election.cc:302] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a53874d1fd84f1b91162b6a76521cdd, bbd4229fd21f4fe1a8700322ad3587a8; no voters:
I20260806 04:30:22.939742 11119 raft_consensus.cc:2781] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:30:22.939824 11119 raft_consensus.cc:482] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:30:22.939853 11119 raft_consensus.cc:3037] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:22.940979 11119 raft_consensus.cc:504] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.941337 11119 leader_election.cc:288] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [CANDIDATE]: Term 1 election: Requested vote from peers bbd4229fd21f4fe1a8700322ad3587a8 (127.28.94.1:38629), 396e6d276d8042e6b1bf397922a18574 (127.28.94.3:40159)
I20260806 04:30:22.941785 10624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fffa47fd545432593f92076f534b733" candidate_uuid: "4a53874d1fd84f1b91162b6a76521cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd4229fd21f4fe1a8700322ad3587a8"
I20260806 04:30:22.941831 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fffa47fd545432593f92076f534b733" candidate_uuid: "4a53874d1fd84f1b91162b6a76521cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "396e6d276d8042e6b1bf397922a18574"
I20260806 04:30:22.941921 10624 raft_consensus.cc:3037] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:22.941946 11003 raft_consensus.cc:3037] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:30:22.943032 11003 raft_consensus.cc:2454] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a53874d1fd84f1b91162b6a76521cdd in term 1.
I20260806 04:30:22.943035 10624 raft_consensus.cc:2454] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a53874d1fd84f1b91162b6a76521cdd in term 1.
I20260806 04:30:22.943313 10723 leader_election.cc:302] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a53874d1fd84f1b91162b6a76521cdd, bbd4229fd21f4fe1a8700322ad3587a8; no voters:
I20260806 04:30:22.943509 11119 raft_consensus.cc:2781] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:30:22.943744 11119 raft_consensus.cc:686] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 1 LEADER]: Becoming Leader. State: Replica: 4a53874d1fd84f1b91162b6a76521cdd, State: Running, Role: LEADER
I20260806 04:30:22.944073 11119 consensus_queue.cc:237] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } }
I20260806 04:30:22.947868 10431 catalog_manager.cc:5495] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a53874d1fd84f1b91162b6a76521cdd (127.28.94.2). New cstate: current_term: 1 leader_uuid: "4a53874d1fd84f1b91162b6a76521cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a53874d1fd84f1b91162b6a76521cdd" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 44481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 } health_report { overall_health: UNKNOWN } } }
I20260806 04:30:22.969810 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:22.966255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.28.94.2:44481)
0806 04:30:22.966398 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:30:22.966402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:30:22.966417 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:30:22.966521 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:30:22.966523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:30:22.966581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:30:22.966675 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:30:22.966923 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.967434 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.968276 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:30:22.968488 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:30:22.968573 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:30:22.968885 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:30:22.969420 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:30:22.969534 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:30:22.969601 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 04:30:22.969664 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:30:22.976135 10624 raft_consensus.cc:1261] T 9fffa47fd545432593f92076f534b733 P bbd4229fd21f4fe1a8700322ad3587a8 [term 1 FOLLOWER]: Refusing update from remote peer 4a53874d1fd84f1b91162b6a76521cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:30:22.976133 11003 raft_consensus.cc:1261] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 1 FOLLOWER]: Refusing update from remote peer 4a53874d1fd84f1b91162b6a76521cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:30:22.976958 11126 consensus_queue.cc:1035] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "396e6d276d8042e6b1bf397922a18574" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:30:22.977210 11119 consensus_queue.cc:1035] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd4229fd21f4fe1a8700322ad3587a8" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:30:22.995371 10431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:30:22.995589 10431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:30:22.998476 10431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54db37d955f54c1aa9ba466cee40f557: Sending DeleteTablet for 3 replicas of tablet 9fffa47fd545432593f92076f534b733
I20260806 04:30:22.999429 10796 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fffa47fd545432593f92076f534b733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:30:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260806 04:30:22.999698 10593 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fffa47fd545432593f92076f534b733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:30:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:33844
I20260806 04:30:23.000073 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10512
I20260806 04:30:23.000265 10977 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fffa47fd545432593f92076f534b733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:30:22 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:45928
I20260806 04:30:23.009371 11144 tablet_replica.cc:330] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: stopping tablet replica
I20260806 04:30:23.009892 11144 raft_consensus.cc:2229] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:30:23.010452 11144 raft_consensus.cc:2258] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:30:23.012609 11144 ts_tablet_manager.cc:1905] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:30:23.015776 11144 ts_tablet_manager.cc:1918] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:30:23.015918 11144 log.cc:1196] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/9fffa47fd545432593f92076f534b733
I20260806 04:30:23.016266 11144 ts_tablet_manager.cc:1939] T 9fffa47fd545432593f92076f534b733 P 396e6d276d8042e6b1bf397922a18574: Deleting consensus metadata
I20260806 04:30:23.016297 11140 tablet_replica.cc:330] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: stopping tablet replica
I20260806 04:30:23.016824 11140 raft_consensus.cc:2229] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:30:23.017374 11140 raft_consensus.cc:2258] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:30:23.017921 10413 catalog_manager.cc:4841] TS 396e6d276d8042e6b1bf397922a18574 (127.28.94.3:40159): tablet 9fffa47fd545432593f92076f534b733 (table test-table [id=b500e03e878e4767b5ad2fd3480445b8]) successfully deleted
I20260806 04:30:23.019244 11140 ts_tablet_manager.cc:1905] T 9fffa47fd545432593f92076f534b733 P 4a53874d1fd84f1b91162b6a76521cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:30:23.020535 10413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:38629 (error 108)
I20260806 04:30:23.021672 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:23.021370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:38629 (local address 127.0.0.1:33856)
0806 04:30:23.021509 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:23.021579 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:38629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:30:23.021823 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10699
W20260806 04:30:23.022007 10413 catalog_manager.cc:4584] TS bbd4229fd21f4fe1a8700322ad3587a8 (127.28.94.1:38629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fffa47fd545432593f92076f534b733: Network error: Client connection negotiation failed: client connection to 127.28.94.1:38629: connect: Connection refused (error 111)
I20260806 04:30:23.033357 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10884
I20260806 04:30:23.039486 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:30:23.033575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:44481 (local address 127.0.0.1:60344)
0806 04:30:23.039266 (+ 5691us) negotiation.cc:107] Waiting for socket to connect
0806 04:30:23.039362 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.2:44481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5545}
I20260806 04:30:23.046419 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 10391
2026-08-06T04:30:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2439 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:30:23 dist-test-slave-ncp9 krb5kdc[11149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:30:23 dist-test-slave-ncp9 krb5kdc[11149](info): set up 2 sockets
Aug 06 04:30:23 dist-test-slave-ncp9 krb5kdc[11149](info): commencing operation
krb5kdc: starting...
W20260806 04:30:24.157413 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:30:24 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:30:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangeradmin_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangeradmin_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangerlookup_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangerlookup_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
I20260806 04:30:24.339974 29048 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4FkAn1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/postgres -l logfile start
2026-08-06 04:30:26.795 UTC [11188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:30:26.795 UTC [11188] LOG: listening on IPv4 address "127.28.94.20", port 44279
2026-08-06 04:30:26.798 UTC [11188] LOG: listening on Unix socket "/tmp/.s.PGSQL.44279"
2026-08-06 04:30:26.819 UTC [11191] LOG: database system was shut down at 2026-08-06 04:30:25 UTC
2026-08-06 04:30:26.823 UTC [11188] LOG: database system is ready to accept connections
I20260806 04:30:27.835948 29048 mini_postgres.cc:96] Postgres bound to 44279
2026-08-06 04:30:27.845 UTC [11200] FATAL: database "slave" does not exist
127.28.94.20:44279 - accepting connections
I20260806 04:30:27.846017 29048 mini_ranger.cc:161] Starting Ranger...
I20260806 04:30:27.863561 29048 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:30:28.292771 29048 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:30:28.292855 29048 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin
2026-08-06 04:30:29,878 [I] DB FLAVOR :POSTGRES
2026-08-06 04:30:29,878 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:30:29,878 [I] Checking connection..
2026-08-06 04:30:29,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:30:30,156 [I] Checking connection passed.
2026-08-06 04:30:30,156 [I] --------- Verifying version history table ---------
2026-08-06 04:30:30,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:30:30,435 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:30:30,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:30:30,701 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:30:30,702 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:30:30,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:30:30.968 UTC [11291] WARNING: there is no transaction in progress
2026-08-06 04:30:30,977 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:30:30,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:30:31,250 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:30:31,250 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:30:31,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:30:31,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:30:31,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-06 04:30:32,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:30:32,281 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:30:32,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:30:32,523 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:30:32,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:30:32,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:30:33,052 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:30:33,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:30:34.029 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.046 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.062 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.078 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.094 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.112 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.167 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.174 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.189 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.200 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.210 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.224 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.233 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.240 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.246 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34.252 UTC [11440] WARNING: there is no transaction in progress
2026-08-06 04:30:34,872 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:30:34,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:30:35,122 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:30:35,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:30:35,396 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:30:35,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:30:35,657 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:30:35,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:30:35,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:30:36,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-06 04:30:36,450 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:30:36,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:30:36,709 [I] DB_PATCHES have already been applied
I20260806 04:30:36.715538 29048 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4FkAn1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:30:36.715641 29048 mini_ranger.cc:193] Using host: 127.28.94.20
I20260806 04:30:36.720682 29048 mini_ranger.cc:227] Ranger admin URL: http://127.28.94.20:60371
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:30:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:30:37.920063 29048 mini_ranger.cc:160] Time spent starting Ranger: real 10.074s user 0.003s sys 0.010s
Aug 06, 2026 4:30:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:30:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 4:30:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:30:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:30:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:31:00.581494 29048 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangerkms_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/rangerkms_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.94.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.94.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:31:00 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990660, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:31:00.665423 29048 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:31:00.675732 29048 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:31:01.417800 29048 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:31:01.417886 29048 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms
I20260806 04:31:01.417964 29048 mini_ranger_kms.cc:227] Using postgres at 127.28.94.20:44279
2026-08-06 04:31:01,968 [I] DB FLAVOR :POSTGRES
2026-08-06 04:31:01,968 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:31:01,968 [I] Checking connection
2026-08-06 04:31:01,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:31:02,243 [I] connection success
2026-08-06 04:31:02,243 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:31:02,243 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:31:02,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:31:02,480 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:31:02,480 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:31:02,480 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:31:02,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:44279/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:31:02,747 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:31:03.209471 29048 mini_ranger_kms.cc:310] Created kms service
I20260806 04:31:03.392035 29048 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:31:03.436633 29048 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:31:03 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990660, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:03.899014 29048 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:31:03.899199 29048 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms:/tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:31:03.899230 29048 mini_ranger_kms.cc:240] Using host: 127.28.94.20
I20260806 04:31:03.905346 29048 mini_ranger_kms.cc:276] Ranger KMS PID: 11685
I20260806 04:31:03.905473 29048 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.94.20:40257
Aug 06, 2026 4:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:31:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:31:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:31:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 04:31:06 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785990666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:06.314404 29048 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.649s user 0.034s sys 0.298s
I20260806 04:31:06.314548 29048 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:31:06.314674 29048 mini_ranger_kms.cc:401] 127.28.94.20:40257/kms/v1/keys
Aug 06 04:31:06 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
Aug 06 04:31:07 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
Aug 06 04:31:09 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:31:09.391564 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36105
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:36105
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:40257/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:09.514242 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:09.514499 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:09.514542 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:09.518496 11755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:31:09.518555 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:09.518573 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:09.518591 11755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:31:09.518621 11755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:31:09.524049 11755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.94.20:40257/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:36105
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:09.525117 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:09.526151 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:09.534750 11766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:09.534772 11760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:09.534760 11761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:09.534868 11755 server_base.cc:1034] running on GCE node
I20260806 04:31:09.535624 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:09.536733 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:09.537925 11755 hybrid_clock.cc:648] HybridClock initialized: now 1785990669537895 us; error 40 us; skew 500 ppm
Aug 06 04:31:09 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990669, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:09.540659 11755 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:31:09.541584 11755 webserver.cc:460] Webserver started at http://127.28.94.62:38049/ using document root <none> and password file <none>
I20260806 04:31:09.542122 11755 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:09.542179 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:09.542388 11755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:09 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990669, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:09.597530 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "59e5ee5a4d2b4c65a31c1350495309a8"
format_stamp: "Formatted at 2026-08-06 04:31:09 on dist-test-slave-ncp9"
server_key: "9a54ad9ce0257f5d94b593c4746abfd4"
server_key_iv: "04e242a667c5cfb6ea2a4a5fd704b59b"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:09.598166 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "59e5ee5a4d2b4c65a31c1350495309a8"
format_stamp: "Formatted at 2026-08-06 04:31:09 on dist-test-slave-ncp9"
server_key: "9a54ad9ce0257f5d94b593c4746abfd4"
server_key_iv: "04e242a667c5cfb6ea2a4a5fd704b59b"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:09.602013 11755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:31:09.616844 11776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:09.618323 11755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:31:09.618465 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "59e5ee5a4d2b4c65a31c1350495309a8"
format_stamp: "Formatted at 2026-08-06 04:31:09 on dist-test-slave-ncp9"
server_key: "9a54ad9ce0257f5d94b593c4746abfd4"
server_key_iv: "04e242a667c5cfb6ea2a4a5fd704b59b"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:09.618597 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:09.642002 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:09.645172 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:09.645375 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:09.653764 11755 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:36105
I20260806 04:31:09.653808 11847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:36105 every 8 connection(s)
I20260806 04:31:09.654798 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:31:09.658528 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:31:09.660449 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 11755
I20260806 04:31:09.660570 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:31:09.660851 29048 external_mini_cluster.cc:1428] Setting key b07e87b6ca0f5577be9fb9ee5e4095fe
I20260806 04:31:09.666076 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: Bootstrap starting.
I20260806 04:31:09.668943 11848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: Neither blocks nor log segments found. Creating new log.
Aug 06 04:31:09 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:09.669922 11848 log.cc:824] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:09.672710 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: No bootstrap required, opened a new log
I20260806 04:31:09.674940 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:09.662329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.28.94.62:36105)
0806 04:31:09.662749 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:09.662764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:09.662801 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:31:09.663459 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:09.663476 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:09.663817 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:09.664120 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:09.664952 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:09.665952 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:09.666876 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:09.667339 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:09.667484 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:09.669916 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:09.669921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:09.669934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:09.669962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:09.672075 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:09.672496 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:09.672501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:09.672506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:09.672612 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:09.672917 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:09.672921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:09.672922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:09.673324 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:09.673528 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:09.673795 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 04:31:09.674208 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260806 04:31:09.677004 11848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } }
I20260806 04:31:09.677294 11848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:31:09.677352 11848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e5ee5a4d2b4c65a31c1350495309a8, State: Initialized, Role: FOLLOWER
I20260806 04:31:09.677825 11848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } }
I20260806 04:31:09.677986 11848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:31:09.678056 11848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:31:09.678150 11848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:31:09.679347 11848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } }
I20260806 04:31:09.679723 11848 leader_election.cc:302] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e5ee5a4d2b4c65a31c1350495309a8; no voters:
I20260806 04:31:09.680107 11848 leader_election.cc:288] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:31:09.680217 11857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:31:09.680459 11857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [term 1 LEADER]: Becoming Leader. State: Replica: 59e5ee5a4d2b4c65a31c1350495309a8, State: Running, Role: LEADER
I20260806 04:31:09.680814 11857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } }
I20260806 04:31:09.681532 11848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:31:09.681946 11857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e5ee5a4d2b4c65a31c1350495309a8. Latest consensus state: current_term: 1 leader_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } } }
I20260806 04:31:09.682066 11857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:09.682586 11859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e5ee5a4d2b4c65a31c1350495309a8" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36105 } } }
I20260806 04:31:09.682694 11859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:09.683991 11848 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:31:09.685408 11865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:31:09.691334 11865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:31:09.699167 11865 catalog_manager.cc:1347] Generated new cluster ID: fb67d7c155dc4e75b78161bc8963ce17
I20260806 04:31:09.699247 11865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:31:09.717186 11865 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:31:09.718650 11865 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:31:09.729094 11865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: Generated new TSK 0
I20260806 04:31:09.730069 11865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:31:10.415452 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:40257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:10.586421 11921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:10.586820 11921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:10.586882 11921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:10.592420 11921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:10.592577 11921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:10.592711 11921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:31:10.599277 11921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.94.20:40257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:10.600459 11921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:10.601588 11921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:10.611637 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:10.611724 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:10.611820 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:10.612715 11921 server_base.cc:1034] running on GCE node
I20260806 04:31:10.613178 11921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:10.613873 11921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:10.615058 11921 hybrid_clock.cc:648] HybridClock initialized: now 1785990670615030 us; error 38 us; skew 500 ppm
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:10.618053 11921 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:31:10.619046 11921 webserver.cc:460] Webserver started at http://127.28.94.1:35635/ using document root <none> and password file <none>
I20260806 04:31:10.619635 11921 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:10.619709 11921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:10.619925 11921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:10.639464 11921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "b478a17588b7443d9d36df411e020b6c"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "387189151274c1c1415d3af0b3a299be"
server_key_iv: "b32150593fd15b0eb840bfd06404b845"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:10.642463 11921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "b478a17588b7443d9d36df411e020b6c"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "387189151274c1c1415d3af0b3a299be"
server_key_iv: "b32150593fd15b0eb840bfd06404b845"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:10.652541 11921 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260806 04:31:10.677877 11942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:10.679860 11921 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260806 04:31:10.680032 11921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "b478a17588b7443d9d36df411e020b6c"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "387189151274c1c1415d3af0b3a299be"
server_key_iv: "b32150593fd15b0eb840bfd06404b845"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:10.680195 11921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:10.722254 11921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:10.727936 11921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:10.728231 11921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:10.729137 11921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:10.730374 11921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:10.730484 11921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:10.730631 11921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:10.730710 11921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:10.749398 11921 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:46455
I20260806 04:31:10.749424 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:46455 every 8 connection(s)
I20260806 04:31:10.750775 11921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:31:10.759747 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 11921
I20260806 04:31:10.759903 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:31:10.760255 29048 external_mini_cluster.cc:1428] Setting key 125ba33f385eebeb6b7710da9988b394
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:10.770984 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:10.753986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36105 (local address 127.28.94.1:51925)
0806 04:31:10.754652 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:10.754713 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:10.756152 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:10.756805 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:10.756815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:10.757287 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:10.757990 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:10.758012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:10.759202 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:10.759206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:10.760634 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:10.760648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:10.761707 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:10.761711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:10.761917 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:10.762694 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:10.762717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:10.764570 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:10.766949 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:10.766956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:10.766986 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:10.767338 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:10.767673 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:10.767677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:10.767679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:10.767868 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:10.768399 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:10.768407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:10.769215 (+ 808us) client_negotiation.cc:769] Sending connection context
0806 04:31:10.769444 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:31:10.769814 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":191,"threads_started":1}
I20260806 04:31:10.771576 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:10.755288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:51925 (local address 127.28.94.62:36105)
0806 04:31:10.755594 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:10.755599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:10.755619 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:31:10.756410 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:10.756415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:10.756475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:10.756591 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:10.758194 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:10.759040 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:10.760788 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:10.761391 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:10.761500 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:10.764790 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:10.764794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:10.764797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:10.764829 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:10.766797 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:10.767462 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:10.767466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:10.767467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:10.767569 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:10.767995 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:10.767999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:10.768000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:10.768249 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:10.768343 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:10.771268 (+ 2925us) server_negotiation.cc:299] Negotiation successful
0806 04:31:10.771408 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260806 04:31:10.773597 12097 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36105
I20260806 04:31:10.773901 12097 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:10.774590 12097 heartbeater.cc:511] Master 127.28.94.62:36105 requested a full tablet report, sending...
I20260806 04:31:10.776320 11799 ts_manager.cc:194] Registered new tserver with Master: b478a17588b7443d9d36df411e020b6c (127.28.94.1:46455)
I20260806 04:31:10.777827 11799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:51925
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:31:10.804054 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:40257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:10.940519 12105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:10.940796 12105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:10.940855 12105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:10.945859 12105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:10.945935 12105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:10.946017 12105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:31:10.951594 12105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.94.20:40257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:10.953053 12105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:10.954495 12105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:10.967707 12113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:10.967707 12115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:10.968092 12112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:10.968781 12105 server_base.cc:1034] running on GCE node
I20260806 04:31:10.969280 12105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:10.969956 12105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:10.971168 12105 hybrid_clock.cc:648] HybridClock initialized: now 1785990670971149 us; error 30 us; skew 500 ppm
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:10.974014 12105 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:31:10.974984 12105 webserver.cc:460] Webserver started at http://127.28.94.2:37581/ using document root <none> and password file <none>
I20260806 04:31:10.975587 12105 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:10.975656 12105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:10.975899 12105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:10 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:10.998665 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "7bd42eea7db44298816c8512123bde1e"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "5c150b32e0d5406fd9dc7688ef7db3ca"
server_key_iv: "124ecc09ad01f3427889d68a917393f4"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.002257 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd42eea7db44298816c8512123bde1e"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "5c150b32e0d5406fd9dc7688ef7db3ca"
server_key_iv: "124ecc09ad01f3427889d68a917393f4"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.010496 12105 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260806 04:31:11.041453 12123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.043401 12105 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20260806 04:31:11.043684 12105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "7bd42eea7db44298816c8512123bde1e"
format_stamp: "Formatted at 2026-08-06 04:31:10 on dist-test-slave-ncp9"
server_key: "5c150b32e0d5406fd9dc7688ef7db3ca"
server_key_iv: "124ecc09ad01f3427889d68a917393f4"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.043960 12105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:11.135154 12105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:11.139063 12105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:11.139320 12105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:11.140101 12105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:11.141386 12105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:11.141471 12105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.141561 12105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:11.141600 12105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.160313 12105 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:35097
I20260806 04:31:11.161681 12105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:31:11.167440 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 12105
I20260806 04:31:11.167606 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:31:11.167963 29048 external_mini_cluster.cc:1428] Setting key 763f2118caff6a45f3f65ca2c55799e0
I20260806 04:31:11.169709 12270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:35097 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:11.205160 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.173691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:33913 (local address 127.28.94.62:36105)
0806 04:31:11.173839 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.173845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.181710 (+ 7865us) server_negotiation.cc:407] Connection header received
0806 04:31:11.186648 (+ 4938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.186653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.186738 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.186810 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:11.188978 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.189828 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.192345 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.192715 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.192857 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.199754 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:11.199760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:11.199763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:11.199807 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:11.201986 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.202897 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.202902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.202904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.203016 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.203537 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.203541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.203543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.203820 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:11.203915 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.204764 (+ 849us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.204970 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:31:11.207442 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.172707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36105 (local address 127.28.94.2:33913)
0806 04:31:11.181355 (+ 8648us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:11.181488 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:11.183187 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:11.187394 (+ 4207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:11.187408 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:11.187887 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:11.188706 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.188732 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.190054 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.190058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.191592 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.191608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.193127 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.193134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.193389 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.194245 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:11.194274 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:11.199414 (+ 5140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:11.202229 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:11.202239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:11.202264 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:11.202747 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:11.203150 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:11.203153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:11.203155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:11.203391 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:11.204000 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:11.204010 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:11.204413 (+ 403us) client_negotiation.cc:769] Sending connection context
0806 04:31:11.205816 (+ 1403us) client_negotiation.cc:240] Negotiation successful
0806 04:31:11.206238 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8300,"thread_start_us":181,"threads_started":1}
I20260806 04:31:11.209235 12272 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36105
I20260806 04:31:11.209553 12272 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:11.210232 12272 heartbeater.cc:511] Master 127.28.94.62:36105 requested a full tablet report, sending...
I20260806 04:31:11.211699 11799 ts_manager.cc:194] Registered new tserver with Master: 7bd42eea7db44298816c8512123bde1e (127.28.94.2:35097)
I20260806 04:31:11.212527 11799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:33913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:31:11.234886 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:40257/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
W20260806 04:31:11.469537 12280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:11.469995 12280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:11.470101 12280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:11.476202 12280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:11.476411 12280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:11.476552 12280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:31:11.484592 12280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.94.20:40257/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:11.486375 12280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:11.487821 12280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:11.501246 12288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:11.501780 12286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:11.502153 12285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:11.502283 12280 server_base.cc:1034] running on GCE node
I20260806 04:31:11.502790 12280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:11.503584 12280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:11.504781 12280 hybrid_clock.cc:648] HybridClock initialized: now 1785990671504750 us; error 44 us; skew 500 ppm
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990671, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:11.508441 12280 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:31:11.509864 12280 webserver.cc:460] Webserver started at http://127.28.94.3:42205/ using document root <none> and password file <none>
I20260806 04:31:11.510651 12280 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:11.510754 12280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:11.511041 12280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990671, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:11.529482 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56"
format_stamp: "Formatted at 2026-08-06 04:31:11 on dist-test-slave-ncp9"
server_key: "1fdb3b94cc998c21574c9be0ed4be629"
server_key_iv: "5c64994617a4aeb13c564c8402a9bdaa"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.530277 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56"
format_stamp: "Formatted at 2026-08-06 04:31:11 on dist-test-slave-ncp9"
server_key: "1fdb3b94cc998c21574c9be0ed4be629"
server_key_iv: "5c64994617a4aeb13c564c8402a9bdaa"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.536264 12280 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 04:31:11.561401 12296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.563344 12280 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260806 04:31:11.563612 12280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56"
format_stamp: "Formatted at 2026-08-06 04:31:11 on dist-test-slave-ncp9"
server_key: "1fdb3b94cc998c21574c9be0ed4be629"
server_key_iv: "5c64994617a4aeb13c564c8402a9bdaa"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:11.563849 12280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:11.590556 12280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:11.594906 12280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:11.595208 12280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:11.596053 12280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:11.597302 12280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:11.597396 12280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.597470 12280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:11.597508 12280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:11.616622 12280 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:37991
I20260806 04:31:11.616642 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:37991 every 8 connection(s)
I20260806 04:31:11.618018 12280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:31:11.627753 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 12280
I20260806 04:31:11.627885 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:31:11.628178 29048 external_mini_cluster.cc:1428] Setting key 35f111bee6b3a60b7d66b1cac761cc03
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990671, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:11.639112 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.621163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:58727 (local address 127.28.94.62:36105)
0806 04:31:11.621303 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.621308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.622091 (+ 783us) server_negotiation.cc:407] Connection header received
0806 04:31:11.623663 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.623667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.623734 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.623793 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:11.625584 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.626399 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.630966 (+ 4567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.631317 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.631393 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.634755 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:11.634759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:11.634762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:11.634794 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:11.636443 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.637197 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.637201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.637203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.637320 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.637727 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.637730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.637732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.637991 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:11.638114 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.638777 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.638922 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:31:11.640327 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.621249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36105 (local address 127.28.94.3:58727)
0806 04:31:11.621923 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:11.621984 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:11.623326 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:11.624263 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:11.624273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:11.624648 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:11.625403 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.625424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.626559 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.626563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.630825 (+ 4262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.630841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.631613 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.631617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.631851 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.632617 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:11.632640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:11.634536 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:11.636583 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:11.636590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:11.636613 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:11.637056 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:11.637423 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:11.637426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:11.637428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:11.637616 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:11.638096 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:11.638103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:11.638473 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 04:31:11.638681 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 04:31:11.639044 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":236,"threads_started":1}
I20260806 04:31:11.643428 12450 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36105
I20260806 04:31:11.643656 12450 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:11.644289 12450 heartbeater.cc:511] Master 127.28.94.62:36105 requested a full tablet report, sending...
I20260806 04:31:11.645717 11799 ts_manager.cc:194] Registered new tserver with Master: ff3bcd05cc0c4e71b21f7608c35b6e56 (127.28.94.3:37991)
I20260806 04:31:11.646451 11799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:58727
I20260806 04:31:11.654752 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:31:11.672044 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.657214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.28.94.62:36105)
0806 04:31:11.657334 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.657340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.661276 (+ 3936us) server_negotiation.cc:407] Connection header received
0806 04:31:11.661380 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.661384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.661442 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.661535 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:11.662450 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.663250 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.664374 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.664700 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.664806 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.666474 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:11.666478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:11.666481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:11.666510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:11.668695 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.669215 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.669219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.669221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.669327 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:11.670666 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:11.670670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:11.670672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:11.670929 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:11.671056 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.671585 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.671731 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 04:31:11 dist-test-slave-ncp9 krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990670, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:11.676177 11799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:31:11.680495 11799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:31:11.726439 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.719676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:35097 (local address 127.0.0.1:50636)
0806 04:31:11.721336 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:11.721349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:11.721789 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:11.722365 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:11.722372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:11.722398 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:31:11.722557 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.722581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.723861 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.723865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.725036 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.725048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.725866 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.725869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.725984 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.726025 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:31:11.726146 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 04:31:11.726283 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"thread_start_us":84,"threads_started":1}
I20260806 04:31:11.726464 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.720264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.28.94.2:35097)
0806 04:31:11.720556 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.720563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.721750 (+ 1187us) server_negotiation.cc:407] Connection header received
0806 04:31:11.721926 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.721933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.722185 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.722448 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:31:11.722690 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.723609 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.725166 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.725740 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.725842 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.726012 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.726190 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.726298 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260806 04:31:11.730558 12173 tablet_service.cc:1467] Processing CreateTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 (DEFAULT_TABLE table=test-table [id=23cc08ebe3cc4ff68c3bcf2c2f05af5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:31:11.732051 12173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774d96a0cf5d4e96bccf25bd0552ee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:31:11.742787 12475 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Bootstrap starting.
I20260806 04:31:11.745755 12475 tablet_bootstrap.cc:654] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Neither blocks nor log segments found. Creating new log.
I20260806 04:31:11.746922 12475 log.cc:824] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:11.749481 12475 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: No bootstrap required, opened a new log
I20260806 04:31:11.749728 12475 ts_tablet_manager.cc:1397] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 04:31:11.756327 12475 raft_consensus.cc:348] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.756824 12475 raft_consensus.cc:374] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:31:11.756958 12475 raft_consensus.cc:729] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd42eea7db44298816c8512123bde1e, State: Initialized, Role: FOLLOWER
I20260806 04:31:11.757340 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.719968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46455 (local address 127.0.0.1:44464)
0806 04:31:11.721580 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:11.721591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:11.721663 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:11.736871 (+ 15208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:11.736874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:11.736926 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:31:11.737046 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.737054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.742261 (+ 5207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.742265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.744777 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.744794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.756307 (+ 11513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.756312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.756955 (+ 643us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.757006 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:31:11.757076 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:31:11.757178 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"thread_start_us":105,"threads_started":1}
I20260806 04:31:11.757913 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.720761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.28.94.1:46455)
0806 04:31:11.733702 (+ 12941us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.733709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.733734 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:31:11.733811 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.733816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.734779 (+ 963us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.734926 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:31:11.738269 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.740739 (+ 2470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.746015 (+ 5276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.747040 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.747156 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.747316 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.757641 (+ 10325us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.757779 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":124,"threads_started":1}
I20260806 04:31:11.757990 12475 consensus_queue.cc:260] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.759127 12475 ts_tablet_manager.cc:1428] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:31:11.761085 12272 heartbeater.cc:503] Master 127.28.94.62:36105 was elected leader, sending a full tablet report...
I20260806 04:31:11.761626 12009 tablet_service.cc:1467] Processing CreateTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 (DEFAULT_TABLE table=test-table [id=23cc08ebe3cc4ff68c3bcf2c2f05af5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:31:11.762884 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774d96a0cf5d4e96bccf25bd0552ee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:31:11.773007 12482 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Bootstrap starting.
I20260806 04:31:11.779681 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.718479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:37991 (local address 127.0.0.1:46304)
0806 04:31:11.719194 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:11.719247 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:11.719375 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:11.734072 (+ 14697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:11.734076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:11.734102 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:31:11.734211 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.734218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.758262 (+ 24044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.758267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.759411 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:11.759424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.779188 (+ 19764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.779194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:11.779314 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.779350 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:31:11.779410 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 04:31:11.779512 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":135,"threads_started":1}
I20260806 04:31:11.780531 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:11.718681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.28.94.3:37991)
0806 04:31:11.732821 (+ 14140us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:11.732832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:11.732861 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:31:11.732973 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:11.732989 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:11.733209 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:11.733374 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:31:11.747859 (+ 14485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.755282 (+ 7423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.777470 (+ 22188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:11.779005 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:11.780034 (+ 1029us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:11.780219 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:11.780280 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 04:31:11.780382 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14008,"thread_start_us":111,"threads_started":1}
I20260806 04:31:11.780956 12097 heartbeater.cc:503] Master 127.28.94.62:36105 was elected leader, sending a full tablet report...
I20260806 04:31:11.784108 12365 tablet_service.cc:1467] Processing CreateTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 (DEFAULT_TABLE table=test-table [id=23cc08ebe3cc4ff68c3bcf2c2f05af5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:31:11.785521 12365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 774d96a0cf5d4e96bccf25bd0552ee60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:31:11.794318 12482 tablet_bootstrap.cc:654] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Neither blocks nor log segments found. Creating new log.
I20260806 04:31:11.795447 12482 log.cc:824] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:11.795534 12483 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Bootstrap starting.
I20260806 04:31:11.798696 12483 tablet_bootstrap.cc:654] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Neither blocks nor log segments found. Creating new log.
I20260806 04:31:11.799727 12483 log.cc:824] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:11.802521 12483 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: No bootstrap required, opened a new log
I20260806 04:31:11.802552 12482 tablet_bootstrap.cc:492] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: No bootstrap required, opened a new log
I20260806 04:31:11.802770 12483 ts_tablet_manager.cc:1397] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260806 04:31:11.802770 12482 ts_tablet_manager.cc:1397] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20260806 04:31:11.807051 12483 raft_consensus.cc:348] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.807427 12483 raft_consensus.cc:374] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:31:11.807535 12483 raft_consensus.cc:729] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3bcd05cc0c4e71b21f7608c35b6e56, State: Initialized, Role: FOLLOWER
I20260806 04:31:11.808139 12483 consensus_queue.cc:260] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.809125 12482 raft_consensus.cc:348] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.809544 12483 ts_tablet_manager.cc:1428] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:31:11.809994 12450 heartbeater.cc:503] Master 127.28.94.62:36105 was elected leader, sending a full tablet report...
I20260806 04:31:11.814675 12482 raft_consensus.cc:374] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:31:11.814815 12482 raft_consensus.cc:729] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b478a17588b7443d9d36df411e020b6c, State: Initialized, Role: FOLLOWER
I20260806 04:31:11.815429 12482 consensus_queue.cc:260] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:11.816783 12482 ts_tablet_manager.cc:1428] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
W20260806 04:31:11.870808 12451 tablet.cc:2369] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:31:11.922551 12273 tablet.cc:2369] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:31:12.004168 12098 tablet.cc:2369] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:31:12.076577 12486 raft_consensus.cc:482] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:31:12.076804 12486 raft_consensus.cc:504] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:12.078070 12486 leader_election.cc:288] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd42eea7db44298816c8512123bde1e (127.28.94.2:35097), b478a17588b7443d9d36df411e020b6c (127.28.94.1:46455)
I20260806 04:31:12.082397 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.078309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:35097 (local address 127.28.94.3:54587)
0806 04:31:12.078525 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:12.078541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:12.078644 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:12.079025 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:12.079028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:12.079043 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:31:12.079138 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:12.079143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.079963 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.079966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:12.081044 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:12.081057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.081948 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.081952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:12.082057 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:12.082076 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:31:12.082141 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:31:12.082252 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":14}
I20260806 04:31:12.082397 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.078463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:54587 (local address 127.28.94.2:35097)
0806 04:31:12.078576 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:12.078581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:12.078597 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:31:12.078782 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:12.078785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:12.078843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:12.078907 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:31:12.079269 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.079849 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.081230 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.081840 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.081938 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:12.082009 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:12.082183 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:31:12.082267 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:31:12.083031 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.078898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:39745 (local address 127.28.94.1:46455)
0806 04:31:12.079016 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:12.079021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:12.079036 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:31:12.079090 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:12.079094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:12.079141 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:12.079228 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:31:12.079492 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.080270 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.081707 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.082299 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.082397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:12.082458 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:12.082829 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 04:31:12.082912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:31:12.083318 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774d96a0cf5d4e96bccf25bd0552ee60" candidate_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd42eea7db44298816c8512123bde1e" is_pre_election: true
I20260806 04:31:12.083524 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.078375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46455 (local address 127.28.94.3:39745)
0806 04:31:12.078800 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:12.078812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:12.078897 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:12.079257 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:12.079259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:12.079276 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:31:12.079386 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:12.079392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.080397 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.080401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:12.081578 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:12.081588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.082587 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.082591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:12.082688 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:12.082712 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:31:12.083245 (+ 533us) client_negotiation.cc:240] Negotiation successful
0806 04:31:12.083339 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":178,"threads_started":1}
I20260806 04:31:12.083704 12209 raft_consensus.cc:2454] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3bcd05cc0c4e71b21f7608c35b6e56 in term 0.
I20260806 04:31:12.084241 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774d96a0cf5d4e96bccf25bd0552ee60" candidate_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b478a17588b7443d9d36df411e020b6c" is_pre_election: true
I20260806 04:31:12.084303 12297 leader_election.cc:302] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd42eea7db44298816c8512123bde1e, ff3bcd05cc0c4e71b21f7608c35b6e56; no voters:
I20260806 04:31:12.084522 12036 raft_consensus.cc:2454] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff3bcd05cc0c4e71b21f7608c35b6e56 in term 0.
I20260806 04:31:12.084661 12486 raft_consensus.cc:2781] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:31:12.084753 12486 raft_consensus.cc:482] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:31:12.084797 12486 raft_consensus.cc:3037] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:31:12.086066 12486 raft_consensus.cc:504] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:12.086427 12486 leader_election.cc:288] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [CANDIDATE]: Term 1 election: Requested vote from peers 7bd42eea7db44298816c8512123bde1e (127.28.94.2:35097), b478a17588b7443d9d36df411e020b6c (127.28.94.1:46455)
I20260806 04:31:12.086796 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774d96a0cf5d4e96bccf25bd0552ee60" candidate_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b478a17588b7443d9d36df411e020b6c"
I20260806 04:31:12.086841 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "774d96a0cf5d4e96bccf25bd0552ee60" candidate_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd42eea7db44298816c8512123bde1e"
I20260806 04:31:12.086930 12036 raft_consensus.cc:3037] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:31:12.086993 12209 raft_consensus.cc:3037] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:31:12.088035 12036 raft_consensus.cc:2454] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3bcd05cc0c4e71b21f7608c35b6e56 in term 1.
I20260806 04:31:12.088325 12209 raft_consensus.cc:2454] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff3bcd05cc0c4e71b21f7608c35b6e56 in term 1.
I20260806 04:31:12.088344 12299 leader_election.cc:302] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b478a17588b7443d9d36df411e020b6c, ff3bcd05cc0c4e71b21f7608c35b6e56; no voters:
I20260806 04:31:12.088521 12486 raft_consensus.cc:2781] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:31:12.088873 12486 raft_consensus.cc:686] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 1 LEADER]: Becoming Leader. State: Replica: ff3bcd05cc0c4e71b21f7608c35b6e56, State: Running, Role: LEADER
I20260806 04:31:12.089298 12486 consensus_queue.cc:237] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } }
I20260806 04:31:12.093024 11799 catalog_manager.cc:5495] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff3bcd05cc0c4e71b21f7608c35b6e56 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff3bcd05cc0c4e71b21f7608c35b6e56" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 } health_report { overall_health: UNKNOWN } } }
I20260806 04:31:12.186699 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.182660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.28.94.3:37991)
0806 04:31:12.182834 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:12.182839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:12.182855 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:31:12.182946 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:12.182949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:12.183013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:12.183132 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:31:12.183386 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.184127 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.185023 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:12.185355 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:12.185455 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:12.185739 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:31:12.186318 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:31:12.186425 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:12.186494 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 04:31:12.186554 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:31:12.192966 12209 raft_consensus.cc:1261] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 1 FOLLOWER]: Refusing update from remote peer ff3bcd05cc0c4e71b21f7608c35b6e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:31:12.192966 12036 raft_consensus.cc:1261] T 774d96a0cf5d4e96bccf25bd0552ee60 P b478a17588b7443d9d36df411e020b6c [term 1 FOLLOWER]: Refusing update from remote peer ff3bcd05cc0c4e71b21f7608c35b6e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:31:12.193781 12501 consensus_queue.cc:1035] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd42eea7db44298816c8512123bde1e" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:31:12.193979 12486 consensus_queue.cc:1035] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b478a17588b7443d9d36df411e020b6c" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:31:12.201588 12505 mvcc.cc:205] Tried to move back new op lower bound from 7315417793294368768 to 7315417792880771072. Current Snapshot: MvccSnapshot[applied={T|T < 7315417793294368768}]
I20260806 04:31:12.202296 12507 mvcc.cc:205] Tried to move back new op lower bound from 7315417793294368768 to 7315417792880771072. Current Snapshot: MvccSnapshot[applied={T|T < 7315417793294368768}]
I20260806 04:31:12.211769 12506 mvcc.cc:205] Tried to move back new op lower bound from 7315417793294368768 to 7315417792880771072. Current Snapshot: MvccSnapshot[applied={T|T < 7315417793294368768}]
I20260806 04:31:12.215222 11793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:31:12.215412 11793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:31:12.217939 11793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59e5ee5a4d2b4c65a31c1350495309a8: Sending DeleteTablet for 3 replicas of tablet 774d96a0cf5d4e96bccf25bd0552ee60
I20260806 04:31:12.218650 12173 tablet_service.cc:1514] Processing DeleteTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:31:12 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:50636
I20260806 04:31:12.218849 12365 tablet_service.cc:1514] Processing DeleteTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:31:12 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:46304
I20260806 04:31:12.218853 12009 tablet_service.cc:1514] Processing DeleteTablet for tablet 774d96a0cf5d4e96bccf25bd0552ee60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:31:12 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:44464
I20260806 04:31:12.220060 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 11921
I20260806 04:31:12.229125 12527 tablet_replica.cc:330] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: stopping tablet replica
I20260806 04:31:12.229539 12527 raft_consensus.cc:2229] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:31:12.230003 12527 raft_consensus.cc:2258] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:31:12.232734 12527 ts_tablet_manager.cc:1905] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:31:12.236714 12527 ts_tablet_manager.cc:1918] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:31:12.236966 12527 log.cc:1196] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/774d96a0cf5d4e96bccf25bd0552ee60
I20260806 04:31:12.237234 12524 tablet_replica.cc:330] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: stopping tablet replica
I20260806 04:31:12.237563 12527 ts_tablet_manager.cc:1939] T 774d96a0cf5d4e96bccf25bd0552ee60 P ff3bcd05cc0c4e71b21f7608c35b6e56: Deleting consensus metadata
I20260806 04:31:12.237617 12524 raft_consensus.cc:2229] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:31:12.238031 12524 raft_consensus.cc:2258] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:31:12.240075 12524 ts_tablet_manager.cc:1905] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:31:12.242930 12524 ts_tablet_manager.cc:1918] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:31:12.243034 12524 log.cc:1196] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/774d96a0cf5d4e96bccf25bd0552ee60
I20260806 04:31:12.244683 11779 catalog_manager.cc:4841] TS ff3bcd05cc0c4e71b21f7608c35b6e56 (127.28.94.3:37991): tablet 774d96a0cf5d4e96bccf25bd0552ee60 (table test-table [id=23cc08ebe3cc4ff68c3bcf2c2f05af5d]) successfully deleted
I20260806 04:31:12.245177 12524 ts_tablet_manager.cc:1939] T 774d96a0cf5d4e96bccf25bd0552ee60 P 7bd42eea7db44298816c8512123bde1e: Deleting consensus metadata
I20260806 04:31:12.246740 11777 catalog_manager.cc:4841] TS 7bd42eea7db44298816c8512123bde1e (127.28.94.2:35097): tablet 774d96a0cf5d4e96bccf25bd0552ee60 (table test-table [id=23cc08ebe3cc4ff68c3bcf2c2f05af5d]) successfully deleted
W20260806 04:31:12.251736 11779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:46455 (error 108)
I20260806 04:31:12.252754 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:12.252464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:46455 (local address 127.0.0.1:44472)
0806 04:31:12.252588 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:12.252663 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:46455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 04:31:12.253073 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 12105
W20260806 04:31:12.253053 11779 catalog_manager.cc:4584] TS b478a17588b7443d9d36df411e020b6c (127.28.94.1:46455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 774d96a0cf5d4e96bccf25bd0552ee60: Network error: Client connection negotiation failed: client connection to 127.28.94.1:46455: connect: Connection refused (error 111)
I20260806 04:31:12.263705 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 12280
I20260806 04:31:12.284009 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 11755
I20260806 04:31:12.299216 29048 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:31:12.643923 29048 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:31:12.644068 29048 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:31:12.988775 29048 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:31:12.989 UTC [11188] LOG: received smart shutdown request
2026-08-06 04:31:12.993 UTC [11188] LOG: background worker "logical replication launcher" (PID 11198) exited with exit code 1
2026-08-06 04:31:12.993 UTC [11192] LOG: shutting down
2026-08-06 04:31:13.136 UTC [11188] LOG: database system is shut down
2026-08-06T04:31:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50182 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:31:13 dist-test-slave-ncp9 krb5kdc[12545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:31:13 dist-test-slave-ncp9 krb5kdc[12545](info): set up 2 sockets
Aug 06 04:31:13 dist-test-slave-ncp9 krb5kdc[12545](info): commencing operation
krb5kdc: starting...
W20260806 04:31:14.344118 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:31:14 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:31:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangeradmin_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangeradmin_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangerlookup_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangerlookup_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
I20260806 04:31:14.512045 29048 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4FkAn1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/postgres -l logfile start
2026-08-06 04:31:16.845 UTC [12586] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:31:16.845 UTC [12586] LOG: listening on IPv4 address "127.28.94.20", port 59327
2026-08-06 04:31:16.847 UTC [12586] LOG: listening on Unix socket "/tmp/.s.PGSQL.59327"
2026-08-06 04:31:16.876 UTC [12592] LOG: database system was shut down at 2026-08-06 04:31:15 UTC
2026-08-06 04:31:16.880 UTC [12586] LOG: database system is ready to accept connections
I20260806 04:31:17.895560 29048 mini_postgres.cc:96] Postgres bound to 59327
2026-08-06 04:31:17.903 UTC [12605] FATAL: database "slave" does not exist
127.28.94.20:59327 - accepting connections
I20260806 04:31:17.903571 29048 mini_ranger.cc:161] Starting Ranger...
I20260806 04:31:17.919448 29048 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:31:18.328320 29048 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:31:18.328409 29048 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin
2026-08-06 04:31:18,611 [I] DB FLAVOR :POSTGRES
2026-08-06 04:31:18,612 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:31:18,612 [I] Checking connection..
2026-08-06 04:31:18,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:31:18,854 [I] Checking connection passed.
2026-08-06 04:31:18,854 [I] --------- Verifying version history table ---------
2026-08-06 04:31:18,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:31:19,093 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:31:19,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:31:19,354 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:31:19,355 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:31:19,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:31:19.611 UTC [12703] WARNING: there is no transaction in progress
2026-08-06 04:31:19,617 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:31:19,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:31:19,880 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:31:19,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:31:19,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:31:20,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:31:20,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ncp9.c.gcp-upstream.internal','N') ;"
2026-08-06 04:31:20,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:31:20,908 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:31:20,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:31:21,147 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:31:21,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:31:21,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:31:21,624 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:31:21,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:31:22.545 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.561 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.577 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.591 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.606 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.622 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.672 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.678 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.692 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.701 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.711 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.723 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.733 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.739 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.745 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:22.750 UTC [12856] WARNING: there is no transaction in progress
2026-08-06 04:31:23,329 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:31:23,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:31:23,567 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:31:23,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:31:23,830 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:31:23,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:31:24,091 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:31:24,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:31:24,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:31:24,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ncp9.c.gcp-upstream.internal';"
2026-08-06 04:31:24,827 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:31:24,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:31:25,092 [I] DB_PATCHES have already been applied
I20260806 04:31:25.099234 29048 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4FkAn1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:31:25.099315 29048 mini_ranger.cc:193] Using host: 127.28.94.20
I20260806 04:31:25.105171 29048 mini_ranger.cc:227] Ranger admin URL: http://127.28.94.20:50343
Aug 06, 2026 4:31:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:31:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:31:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:31:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:31:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:31:26.276803 29048 mini_ranger.cc:160] Time spent starting Ranger: real 8.373s user 0.000s sys 0.012s
Aug 06, 2026 4:31:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:31:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 4:31:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:31:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:31:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:31:47.680873 29048 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.94.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.94.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangerkms_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.94.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/rangerkms_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.94.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.94.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.94.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/HTTP_127.28.94.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:31:47 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:31:47.770946 29048 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:31:47.780033 29048 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:31:48.533301 29048 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:31:48.533378 29048 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms
I20260806 04:31:48.533396 29048 mini_ranger_kms.cc:227] Using postgres at 127.28.94.20:59327
2026-08-06 04:31:48,705 [I] DB FLAVOR :POSTGRES
2026-08-06 04:31:48,705 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:31:48,705 [I] Checking connection
2026-08-06 04:31:48,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:31:48,942 [I] connection success
2026-08-06 04:31:48,942 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:31:48,942 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:31:48,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:31:49,170 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:31:49,170 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:31:49,170 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:31:49,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.94.20:59327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:31:49,426 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:31:49.874643 29048 mini_ranger_kms.cc:310] Created kms service
I20260806 04:31:50.085278 29048 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:31:50.125761 29048 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:31:50 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:50.479981 29048 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:31:50.480136 29048 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms:/tmp/dist-test-task4FkAn1/build/debug/bin/postgresql.jar:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4FkAn1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:31:50.480157 29048 mini_ranger_kms.cc:240] Using host: 127.28.94.20
I20260806 04:31:50.484861 29048 mini_ranger_kms.cc:276] Ranger KMS PID: 13117
I20260806 04:31:50.484969 29048 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.94.20:52883
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:31:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 04:31:52 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785990712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:52.833511 29048 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.063s user 0.029s sys 0.128s
I20260806 04:31:52.833652 29048 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:31:52.833734 29048 mini_ranger_kms.cc:401] 127.28.94.20:52883/kms/v1/keys
Aug 06 04:31:53 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
Aug 06 04:31:53 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.94.20@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
Aug 06 04:31:55 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:31:55.996034 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39379
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39379
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:56.106594 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:56.106855 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:56.106909 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:56.110858 13181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:31:56.110921 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:56.110946 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:56.110967 13181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:31:56.111018 13181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:31:56.116472 13181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.94.20:52883/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39379
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:56.117527 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:56.118542 13181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:56.128373 13189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:56.128546 13181 server_base.cc:1034] running on GCE node
W20260806 04:31:56.128379 13192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:56.128391 13195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:56.129383 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:56.130460 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:56.131614 13181 hybrid_clock.cc:648] HybridClock initialized: now 1785990716131587 us; error 44 us; skew 500 ppm
Aug 06 04:31:56 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990716, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:56.134439 13181 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:31:56.135363 13181 webserver.cc:460] Webserver started at http://127.28.94.62:37323/ using document root <none> and password file <none>
I20260806 04:31:56.135892 13181 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:56.135947 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:56.136149 13181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:56 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990716, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:56.184363 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "1c624c4f8eab439b9e7187e84f85a599"
format_stamp: "Formatted at 2026-08-06 04:31:56 on dist-test-slave-ncp9"
server_key: "4736de96ab822a5879eae6d758a8724f"
server_key_iv: "fb10d7b684dd5762dcfde8bfca60c4dc"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:56.185216 13181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "1c624c4f8eab439b9e7187e84f85a599"
format_stamp: "Formatted at 2026-08-06 04:31:56 on dist-test-slave-ncp9"
server_key: "4736de96ab822a5879eae6d758a8724f"
server_key_iv: "fb10d7b684dd5762dcfde8bfca60c4dc"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:56.190512 13181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:31:56.207335 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:56.208791 13181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:31:56.209007 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "1c624c4f8eab439b9e7187e84f85a599"
format_stamp: "Formatted at 2026-08-06 04:31:56 on dist-test-slave-ncp9"
server_key: "4736de96ab822a5879eae6d758a8724f"
server_key_iv: "fb10d7b684dd5762dcfde8bfca60c4dc"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:56.209154 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:56.260432 13181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:56.263725 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:56.263947 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:56.272264 13181 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39379
I20260806 04:31:56.272276 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39379 every 8 connection(s)
I20260806 04:31:56.273371 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:31:56.273727 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 13181
I20260806 04:31:56.273876 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:31:56.274226 29048 external_mini_cluster.cc:1428] Setting key 6d1cf4bc81a8007253c0ccfd72825865
I20260806 04:31:56.277297 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:31:56 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:56.285034 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Bootstrap starting.
I20260806 04:31:56.287904 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:56.275629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.28.94.62:39379)
0806 04:31:56.276095 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:56.276115 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:56.276392 (+ 277us) server_negotiation.cc:407] Connection header received
0806 04:31:56.277196 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:56.277215 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:56.277591 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:56.277911 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:56.278724 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:56.279720 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:56.280564 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:56.281073 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:56.281231 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:56.283609 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:56.283614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:56.283627 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:56.283655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:56.285194 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:56.285603 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:56.285609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:56.285614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:56.285719 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:56.285998 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:56.286002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:56.286004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:56.286397 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:56.286566 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:56.286877 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 04:31:56.287185 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":163,"threads_started":1}
I20260806 04:31:56.287966 13280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Neither blocks nor log segments found. Creating new log.
I20260806 04:31:56.289407 13280 log.cc:824] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:56.292006 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: No bootstrap required, opened a new log
I20260806 04:31:56.296154 13280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:56.296442 13280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:31:56.296516 13280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c624c4f8eab439b9e7187e84f85a599, State: Initialized, Role: FOLLOWER
I20260806 04:31:56.297025 13280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:56.297197 13280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:31:56.297260 13280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:31:56.297356 13280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:31:56.298525 13280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:56.298941 13280 leader_election.cc:302] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c624c4f8eab439b9e7187e84f85a599; no voters:
I20260806 04:31:56.299345 13280 leader_election.cc:288] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:31:56.299445 13286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:31:56.299685 13286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 LEADER]: Becoming Leader. State: Replica: 1c624c4f8eab439b9e7187e84f85a599, State: Running, Role: LEADER
I20260806 04:31:56.300047 13286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:56.300745 13280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:31:56.301667 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c624c4f8eab439b9e7187e84f85a599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } } }
I20260806 04:31:56.301791 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:56.302158 13289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c624c4f8eab439b9e7187e84f85a599. Latest consensus state: current_term: 1 leader_uuid: "1c624c4f8eab439b9e7187e84f85a599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } } }
I20260806 04:31:56.302322 13289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:56.302660 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:31:56.303138 13280 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:31:56.309428 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:31:56.318504 13301 catalog_manager.cc:1347] Generated new cluster ID: 5877c4a4af5f4d69a3558b9d27e2daa5
I20260806 04:31:56.318567 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:31:56.334353 13301 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:31:56.335821 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:31:56.342983 13301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Generated new TSK 0
I20260806 04:31:56.343932 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:31:57.150457 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:57.310031 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:57.310467 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:57.310599 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:57.316468 13349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:57.316540 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:57.316629 13349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:31:57.324115 13349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:57.325510 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:57.326808 13349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:57.337218 13364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:57.337483 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:57.337642 13349 server_base.cc:1034] running on GCE node
W20260806 04:31:57.337688 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:57.338209 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:57.338971 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:57.340219 13349 hybrid_clock.cc:648] HybridClock initialized: now 1785990717340191 us; error 40 us; skew 500 ppm
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:57.343842 13349 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:31:57.344966 13349 webserver.cc:460] Webserver started at http://127.28.94.1:35767/ using document root <none> and password file <none>
I20260806 04:31:57.345687 13349 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:57.345777 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:57.346065 13349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:57.362812 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "c8017c373fe34b4f9dd18d56ac553425"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "d79c035119f7e6d4971e02c8187ea387"
server_key_iv: "bb4392feff939a1375122098043bfdd0"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.363670 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "c8017c373fe34b4f9dd18d56ac553425"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "d79c035119f7e6d4971e02c8187ea387"
server_key_iv: "bb4392feff939a1375122098043bfdd0"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.368296 13349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 04:31:57.391554 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.393421 13349 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260806 04:31:57.393642 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "c8017c373fe34b4f9dd18d56ac553425"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "d79c035119f7e6d4971e02c8187ea387"
server_key_iv: "bb4392feff939a1375122098043bfdd0"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.393860 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:57.413800 13349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:57.417488 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:57.417754 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:57.418538 13349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:57.419682 13349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:57.419790 13349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.419912 13349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:57.419994 13349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.433475 13349 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:40371
I20260806 04:31:57.435824 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:31:57.439637 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 13349
I20260806 04:31:57.439755 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:31:57.440086 29048 external_mini_cluster.cc:1428] Setting key fdb6297b33ddccfebd3428e2325489ad
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:31:57.470928 13541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:40371 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:57.486718 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:57.470726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:36829 (local address 127.28.94.62:39379)
0806 04:31:57.471356 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:57.471362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:57.473304 (+ 1942us) server_negotiation.cc:407] Connection header received
0806 04:31:57.473966 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:57.473969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:57.474031 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:57.474108 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:57.475631 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.476183 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.477747 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.477988 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.478074 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:57.480802 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:57.480805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:57.480807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:57.480831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:57.482608 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:57.483228 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:57.483232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:57.483233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:57.483330 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:57.483742 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:57.483745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:57.483747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:57.483971 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:57.484048 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:57.484771 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 04:31:57.484931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":102,"threads_started":1}
I20260806 04:31:57.488180 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:57.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.1:36829)
0806 04:31:57.471671 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:57.471735 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:57.473629 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:57.474420 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:57.474429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:57.474765 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:57.475442 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:57.475463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.476313 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.476317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:57.477633 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:57.477645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.478136 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.478141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:57.478327 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:57.478969 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:57.478990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:57.480638 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:57.482740 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:57.482746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:57.482761 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:57.483105 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:57.483444 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:57.483447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:57.483449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:57.483624 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:57.484150 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:57.484157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:57.484493 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 04:31:57.486958 (+ 2465us) client_negotiation.cc:240] Negotiation successful
0806 04:31:57.487309 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1592,"thread_start_us":152,"threads_started":1}
I20260806 04:31:57.489614 13542 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:57.489884 13542 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:57.490479 13542 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:31:57.492213 13230 ts_manager.cc:194] Registered new tserver with Master: c8017c373fe34b4f9dd18d56ac553425 (127.28.94.1:40371)
I20260806 04:31:57.493157 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:31:57.493654 13230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:36829
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 04:31:57.621619 13550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:57.621881 13550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:57.621941 13550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:57.625955 13550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:57.626014 13550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:57.626093 13550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:31:57.632345 13550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:57.633770 13550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:57.634872 13550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:57.645226 13558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:57.645382 13562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:57.645754 13559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:57.646257 13550 server_base.cc:1034] running on GCE node
I20260806 04:31:57.646750 13550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:57.647599 13550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:57.651059 13550 hybrid_clock.cc:648] HybridClock initialized: now 1785990717651000 us; error 71 us; skew 500 ppm
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:57.654959 13550 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:31:57.656286 13550 webserver.cc:460] Webserver started at http://127.28.94.2:38511/ using document root <none> and password file <none>
I20260806 04:31:57.657094 13550 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:57.657269 13550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:57.657600 13550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:57.676178 13550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "407b9f42607740eba783ebc1f7971a53"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "1b1ab8cdbc97f569d4983d05383b212c"
server_key_iv: "cd65638fe258212e7ff87c269326f533"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.677503 13550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "407b9f42607740eba783ebc1f7971a53"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "1b1ab8cdbc97f569d4983d05383b212c"
server_key_iv: "cd65638fe258212e7ff87c269326f533"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.686533 13550 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.004s
I20260806 04:31:57.714038 13573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.715992 13550 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260806 04:31:57.716169 13550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "407b9f42607740eba783ebc1f7971a53"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "1b1ab8cdbc97f569d4983d05383b212c"
server_key_iv: "cd65638fe258212e7ff87c269326f533"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:57.716341 13550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:57.739063 13550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:57.743007 13550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:57.743291 13550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:57.744194 13550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:57.745467 13550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:57.745580 13550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.745689 13550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:57.745762 13550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:57.763962 13550 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:45331
I20260806 04:31:57.765030 13550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:31:57.766261 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:45331 every 8 connection(s)
I20260806 04:31:57.766741 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 13550
I20260806 04:31:57.766852 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:31:57.767176 29048 external_mini_cluster.cc:1428] Setting key 313092e796bddf43feb2172f12110b06
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:31:57.781327 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:57.769525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:59995 (local address 127.28.94.62:39379)
0806 04:31:57.769634 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:57.769638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:57.769653 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:31:57.770454 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:57.770458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:57.770513 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:57.770591 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:57.772093 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.772667 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.773966 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.774204 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.774297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:57.776620 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:57.776623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:57.776625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:57.776652 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:57.778382 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:57.779400 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:57.779403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:57.779405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:57.779511 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:57.779947 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:57.779950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:57.779966 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:57.780205 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:57.780295 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:57.781025 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 04:31:57.781158 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:31:57.782240 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:57.768349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.2:59995)
0806 04:31:57.769022 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:57.769088 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:57.770183 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:57.770805 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:57.770815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:57.771201 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:57.771913 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:57.771936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.772798 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.772801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:57.773844 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:57.773856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:57.774353 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:57.774357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:57.774533 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:57.775086 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:57.775103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:57.776455 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:57.778915 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:57.778921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:57.778938 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:57.779276 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:57.779622 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:57.779626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:57.779628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:57.779827 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:57.780366 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:57.780373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:57.780736 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 04:31:57.780975 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 04:31:57.781293 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":169,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:31:57.783633 13732 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:57.783932 13732 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:57.784583 13732 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:57.785785 13230 ts_manager.cc:194] Registered new tserver with Master: 407b9f42607740eba783ebc1f7971a53 (127.28.94.2:45331)
I20260806 04:31:57.786556 13230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:59995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:31:57.805477 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:57.943221 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:57.943500 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:57.943562 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:57.947805 13737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:57.947875 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:57.947955 13737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:31:57.953845 13737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:57.955015 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:57.956084 13737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:57.966245 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:57.966404 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:57.966244 13742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:57.966638 13737 server_base.cc:1034] running on GCE node
I20260806 04:31:57.967268 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:57.968000 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:57.969170 13737 hybrid_clock.cc:648] HybridClock initialized: now 1785990717969014 us; error 171 us; skew 500 ppm
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:57.972952 13737 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:31:57.974112 13737 webserver.cc:460] Webserver started at http://127.28.94.3:37105/ using document root <none> and password file <none>
I20260806 04:31:57.974692 13737 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:57.974761 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:31:57.975047 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:31:57 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:58.006615 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "12fe8dc3ea004aa4bc637e77105d5494"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "40289ad73b4b8d311476b7dacc00c0af"
server_key_iv: "32f82790fe37fde1e8ab7ff4efa23932"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.007525 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "12fe8dc3ea004aa4bc637e77105d5494"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "40289ad73b4b8d311476b7dacc00c0af"
server_key_iv: "32f82790fe37fde1e8ab7ff4efa23932"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.024760 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:58.053275 13755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.055244 13737 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260806 04:31:58.055397 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "12fe8dc3ea004aa4bc637e77105d5494"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "40289ad73b4b8d311476b7dacc00c0af"
server_key_iv: "32f82790fe37fde1e8ab7ff4efa23932"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.055577 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:58.098616 13737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:58.102407 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:58.102665 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:58.103458 13737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:58.104581 13737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:58.104645 13737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.104696 13737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:58.104723 13737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.119885 13737 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40603
I20260806 04:31:58.119904 13923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40603 every 8 connection(s)
I20260806 04:31:58.121273 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:31:58.126817 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 13737
I20260806 04:31:58.126938 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:31:58.127247 29048 external_mini_cluster.cc:1428] Setting key 6a02b0fd1161a71b3e5c9df0e62aea85
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:58.136682 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.123610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:60631 (local address 127.28.94.62:39379)
0806 04:31:58.123743 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:58.123748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:58.124958 (+ 1210us) server_negotiation.cc:407] Connection header received
0806 04:31:58.126272 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:58.126276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:58.126338 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:58.126437 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:58.127999 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.128820 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.130565 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.130844 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.130905 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:58.133343 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:58.133346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:58.133348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:58.133371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:58.134732 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:58.135202 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:58.135205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:58.135207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:58.135284 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:58.135604 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:58.135608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:58.135610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:58.135790 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:58.135873 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:58.136387 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 04:31:58.136520 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:31:58.137688 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.124145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.3:60631)
0806 04:31:58.124753 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:58.124811 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:58.125965 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:58.126677 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:58.126688 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:58.127083 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:58.127772 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.127792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.128997 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.129001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.130414 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.130426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.131079 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.131082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.131221 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:58.131786 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:58.131803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:58.133152 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:58.134840 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:58.134844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:58.134856 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:58.135097 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:58.135365 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:58.135367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:58.135369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:58.135507 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:58.135886 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:58.135892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:58.136143 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 04:31:58.136309 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 04:31:58.136623 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":193,"threads_started":1}
I20260806 04:31:58.138804 13926 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:58.139045 13926 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:58.139542 13926 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:58.140664 13229 ts_manager.cc:194] Registered new tserver with Master: 12fe8dc3ea004aa4bc637e77105d5494 (127.28.94.3:40603)
I20260806 04:31:58.141433 13229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:60631
I20260806 04:31:58.141470 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:31:58.142349 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 13349
I20260806 04:31:58.153330 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 13550
I20260806 04:31:58.163957 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 13737
I20260806 04:31:58.176396 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 13181
I20260806 04:31:58.189875 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39379
--webserver_interface=127.28.94.62
--webserver_port=37323
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39379
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:31:58.307073 13930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:58.307341 13930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:58.307399 13930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:58.311445 13930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:31:58.311517 13930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:58.311545 13930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:58.311566 13930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:31:58.311591 13930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:31:58.317119 13930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39379
--ranger_config_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:58.318250 13930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:58.319355 13930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:31:58.328094 13930 server_base.cc:1034] running on GCE node
W20260806 04:31:58.328045 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.328045 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.328050 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:58.328929 13930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:58.330116 13930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:58.331276 13930 hybrid_clock.cc:648] HybridClock initialized: now 1785990718331256 us; error 33 us; skew 500 ppm
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:58.334024 13930 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:31:58.334971 13930 webserver.cc:460] Webserver started at http://127.28.94.62:37323/ using document root <none> and password file <none>
I20260806 04:31:58.335541 13930 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:58.335620 13930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:58.375820 13930 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:31:58.378439 13948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.379869 13930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:31:58.380004 13930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "1c624c4f8eab439b9e7187e84f85a599"
format_stamp: "Formatted at 2026-08-06 04:31:56 on dist-test-slave-ncp9"
server_key: "4736de96ab822a5879eae6d758a8724f"
server_key_iv: "fb10d7b684dd5762dcfde8bfca60c4dc"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.380605 13930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:58.407045 13930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:58.410154 13930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:58.410364 13930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:58.417960 13930 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39379
I20260806 04:31:58.417977 14027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39379 every 8 connection(s)
I20260806 04:31:58.419184 13930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:31:58.424117 14028 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:31:58.426105 14028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Bootstrap starting.
I20260806 04:31:58.427943 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 13930
I20260806 04:31:58.428488 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:40371
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:39379
--webserver_port=35767
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:31:58.444684 14028 log.cc:824] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Log is configured to *not* fsync() on all Append() calls
I20260806 04:31:58.452265 14028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:31:58.452694 14028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Bootstrap complete.
I20260806 04:31:58.457049 14028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:58.457486 14028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c624c4f8eab439b9e7187e84f85a599, State: Initialized, Role: FOLLOWER
I20260806 04:31:58.458041 14028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:58.458182 14028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:31:58.458253 14028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:31:58.458349 14028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:31:58.460398 14028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:58.460753 14028 leader_election.cc:302] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c624c4f8eab439b9e7187e84f85a599; no voters:
I20260806 04:31:58.461128 14028 leader_election.cc:288] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:31:58.461251 14032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:31:58.461460 14032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [term 2 LEADER]: Becoming Leader. State: Replica: 1c624c4f8eab439b9e7187e84f85a599, State: Running, Role: LEADER
I20260806 04:31:58.461807 14032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } }
I20260806 04:31:58.462606 14028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:31:58.462924 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c624c4f8eab439b9e7187e84f85a599. Latest consensus state: current_term: 2 leader_uuid: "1c624c4f8eab439b9e7187e84f85a599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } } }
I20260806 04:31:58.463045 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:58.463481 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c624c4f8eab439b9e7187e84f85a599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c624c4f8eab439b9e7187e84f85a599" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39379 } } }
I20260806 04:31:58.463665 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599 [sys.catalog]: This master's current role is: LEADER
I20260806 04:31:58.464926 14038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:31:58.465307 14028 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:31:58.472040 14038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:31:58.473125 14038 catalog_manager.cc:1259] Loaded cluster ID: 5877c4a4af5f4d69a3558b9d27e2daa5
I20260806 04:31:58.473206 14038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:31:58.477216 14038 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:31:58.478204 14038 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Loaded TSK: 0
I20260806 04:31:58.479681 14038 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:31:58.588670 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:58.589107 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:58.589191 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:58.595458 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:58.595559 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:58.595664 14030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:31:58.604101 14030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:40371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=35767
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:58.605794 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:58.607333 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:58.621577 14087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.621767 14088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.622143 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:58.623209 14030 server_base.cc:1034] running on GCE node
I20260806 04:31:58.623698 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:58.624428 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:58.625602 14030 hybrid_clock.cc:648] HybridClock initialized: now 1785990718625565 us; error 54 us; skew 500 ppm
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:58.628654 14030 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:31:58.629788 14030 webserver.cc:460] Webserver started at http://127.28.94.1:35767/ using document root <none> and password file <none>
I20260806 04:31:58.630404 14030 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:58.630496 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:58.648788 14030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:31:58.653607 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.655427 14030 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260806 04:31:58.655582 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "c8017c373fe34b4f9dd18d56ac553425"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "d79c035119f7e6d4971e02c8187ea387"
server_key_iv: "bb4392feff939a1375122098043bfdd0"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.656368 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:58.670806 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:58.674625 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:58.674851 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:58.675657 14030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:58.676842 14030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:58.676944 14030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 04:31:58.677016 14030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:58.677052 14030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.688319 14030 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:40371
I20260806 04:31:58.688480 14280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:40371 every 8 connection(s)
I20260806 04:31:58.689632 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:31:58.698076 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14030
I20260806 04:31:58.698612 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:45331
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:39379
--webserver_port=38511
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:58.717144 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.692646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:46727 (local address 127.28.94.62:39379)
0806 04:31:58.696042 (+ 3396us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:58.696066 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:58.696123 (+ 57us) server_negotiation.cc:407] Connection header received
0806 04:31:58.696425 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:58.696440 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:58.696860 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:58.697261 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:58.702029 (+ 4768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.703160 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.704381 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.704870 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.705057 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:58.707469 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:58.707476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:58.707490 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:58.707520 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:58.709312 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:58.713435 (+ 4123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:58.713443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:58.713450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:58.713627 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:58.714052 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:58.714056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:58.714058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:58.714630 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:58.714769 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:58.715580 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 04:31:58.715918 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3181,"thread_start_us":141,"threads_started":1}
I20260806 04:31:58.718333 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.692747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.1:46727)
0806 04:31:58.694009 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:58.694068 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:58.695159 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:58.700667 (+ 5508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:58.700691 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:58.701091 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:58.701804 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.701829 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.703301 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.703305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.704263 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.704274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.705063 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.705068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.705266 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:58.705854 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:58.705872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:58.707265 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:58.709457 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:58.709463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:58.709480 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:58.713272 (+ 3792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:58.713745 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:58.713749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:58.713751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:58.713943 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:58.714770 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:58.714777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:58.715235 (+ 458us) client_negotiation.cc:769] Sending connection context
0806 04:31:58.715411 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 04:31:58.716165 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":168,"threads_started":1}
I20260806 04:31:58.721032 14281 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:58.721292 14281 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:58.721837 14281 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:58.724885 14281 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:39379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:31:58.725051 14281 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:58.725241 14281 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:58.726058 14281 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:58.726240 14281 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:58.726958 14281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:31:58.852186 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:58.852569 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:58.852659 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:58.857080 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:58.857149 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:58.857247 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:31:58.863031 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:45331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=38511
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:58.864253 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:58.865363 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:58.875905 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.876169 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:58.875880 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:58.876523 14285 server_base.cc:1034] running on GCE node
I20260806 04:31:58.876986 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:58.877731 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:58.878918 14285 hybrid_clock.cc:648] HybridClock initialized: now 1785990718878899 us; error 33 us; skew 500 ppm
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:58.881867 14285 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:31:58.882884 14285 webserver.cc:460] Webserver started at http://127.28.94.2:38511/ using document root <none> and password file <none>
I20260806 04:31:58.883464 14285 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:58.883559 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:58.915118 14285 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260806 04:31:58.921772 14306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.925037 14285 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260806 04:31:58.925556 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "407b9f42607740eba783ebc1f7971a53"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "1b1ab8cdbc97f569d4983d05383b212c"
server_key_iv: "cd65638fe258212e7ff87c269326f533"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:58.926471 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:58.944659 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:58.949748 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:58.950042 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:58.950918 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:58.952103 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:58.952224 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.952334 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:58.952409 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:58.964067 14285 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:45331
I20260806 04:31:58.966173 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:31:58.972040 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14285
I20260806 04:31:58.972596 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.94.20:52883/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:40603
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:39379
--webserver_port=37105
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:31:58.977654 14467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:45331 every 8 connection(s)
Aug 06 04:31:58 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990718, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:59.019912 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.981893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:47551 (local address 127.28.94.62:39379)
0806 04:31:58.982037 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:58.982042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:58.989465 (+ 7423us) server_negotiation.cc:407] Connection header received
0806 04:31:58.990747 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:58.990751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:58.990815 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:58.990899 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:58.992595 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.993443 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.995221 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.995587 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.995675 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:59.001408 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:59.001412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:59.001415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:59.001449 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:59.003310 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.013798 (+ 10488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.013803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.013805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.013923 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.014438 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.014442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.014443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.014714 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:59.014838 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:59.019552 (+ 4714us) server_negotiation.cc:299] Negotiation successful
0806 04:31:59.019729 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:31:59.021873 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:58.980422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.2:47551)
0806 04:31:58.989261 (+ 8839us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:58.989335 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:58.990483 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:58.991184 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:58.991196 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:58.991598 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:58.992362 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.992384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.993639 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.993642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.995069 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:58.995082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:58.995899 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:58.995903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:58.996116 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:58.999286 (+ 3170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:58.999310 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:59.001142 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:59.013091 (+ 11949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:59.013099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:59.013135 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:59.013637 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:59.014059 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:59.014062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:59.014064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:59.014312 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:59.014916 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:59.014924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:59.015332 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 04:31:59.020213 (+ 4881us) client_negotiation.cc:240] Negotiation successful
0806 04:31:59.020647 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8540,"thread_start_us":168,"threads_started":1}
I20260806 04:31:59.023325 14468 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:59.023602 14468 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.024212 14468 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:59.025978 14468 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:39379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:31:59.026117 14468 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.026326 14468 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:59.027192 14468 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.027374 14468 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:59.028116 14468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:31:59.164866 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:31:59.165279 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:31:59.165355 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:31:59.171231 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:31:59.171327 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:31:59.171416 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:31:59.179562 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.94.20:52883/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:40603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=37105
--tserver_master_addrs=127.28.94.62:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:31:59.181103 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:31:59.182720 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:31:59.195902 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:59.196669 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:31:59.195981 14482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:31:59.198758 14470 server_base.cc:1034] running on GCE node
I20260806 04:31:59.199357 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:31:59.200209 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:31:59.201398 14470 hybrid_clock.cc:648] HybridClock initialized: now 1785990719201363 us; error 50 us; skew 500 ppm
Aug 06 04:31:59 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:59.204572 14470 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:31:59.205567 14470 webserver.cc:460] Webserver started at http://127.28.94.3:37105/ using document root <none> and password file <none>
I20260806 04:31:59.206133 14470 fs_manager.cc:359] Metadata directory not provided
I20260806 04:31:59.206218 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:31:59 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:31:59.230331 14470 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260806 04:31:59.233433 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:59.235033 14470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:31:59.235164 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "12fe8dc3ea004aa4bc637e77105d5494"
format_stamp: "Formatted at 2026-08-06 04:31:57 on dist-test-slave-ncp9"
server_key: "40289ad73b4b8d311476b7dacc00c0af"
server_key_iv: "32f82790fe37fde1e8ab7ff4efa23932"
server_key_version: "kuduclusterkey@0"
I20260806 04:31:59.235742 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:31:59.256256 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:31:59.261019 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:31:59.261221 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:31:59.261981 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:31:59.263239 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:31:59.263311 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:59.263374 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:31:59.263404 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:31:59.274971 14470 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40603
I20260806 04:31:59.275000 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40603 every 8 connection(s)
I20260806 04:31:59.276065 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:31:59.283565 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14470
Aug 06 04:31:59 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:31:59.293520 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:59.278542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39379 (local address 127.28.94.3:42557)
0806 04:31:59.279038 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 04:31:59.279091 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:31:59.279995 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:31:59.280554 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:31:59.280562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:31:59.280833 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:31:59.281535 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:59.281557 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.282744 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.282748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:59.284803 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:31:59.284815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.285535 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.285540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:31:59.285758 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:59.286419 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:31:59.286443 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:31:59.287935 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:31:59.289974 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:59.289981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:59.289998 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:59.290333 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:59.290665 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:31:59.290668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:31:59.290670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:31:59.290858 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:31:59.291385 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:31:59.291392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:31:59.291756 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 04:31:59.291992 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 04:31:59.292341 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
I20260806 04:31:59.294384 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:59.278099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:42557 (local address 127.28.94.62:39379)
0806 04:31:59.278239 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:59.278243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:59.279199 (+ 956us) server_negotiation.cc:407] Connection header received
0806 04:31:59.280213 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:59.280217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:59.280280 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:59.280354 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:59.281728 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.282562 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.284942 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.285258 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.285356 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:59.288136 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:59.288140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:59.288143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:59.288169 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:59.289829 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.290441 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.290444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.290446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.290569 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.290991 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.290995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.290997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.291228 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:59.291324 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:59.294104 (+ 2780us) server_negotiation.cc:299] Negotiation successful
0806 04:31:59.294225 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":1280}
I20260806 04:31:59.295922 14653 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39379
I20260806 04:31:59.296169 14653 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.296703 14653 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:59.298487 14653 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:39379 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:31:59.298640 14653 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.298835 14653 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:59.299707 14653 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.299899 14653 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
W20260806 04:31:59.300691 14653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:31:59.301465 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:31:59.284228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.28.94.62:39379)
0806 04:31:59.285064 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0806 04:31:59.285068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:31:59.285313 (+ 245us) server_negotiation.cc:407] Connection header received
0806 04:31:59.285572 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:31:59.285575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:31:59.285666 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:31:59.285728 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:31:59.287609 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.288333 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.290067 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:31:59.290397 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:31:59.290480 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:31:59.292135 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:31:59.292139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:31:59.292142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:31:59.292167 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:31:59.294113 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.294596 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.294599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.294601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.294761 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:31:59.295084 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:31:59.295088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:31:59.295090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:31:59.295595 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:31:59.295765 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 04:31:59.301131 (+ 5366us) server_negotiation.cc:299] Negotiation successful
0806 04:31:59.301276 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"spinlock_wait_cycles":20352,"thread_start_us":94,"threads_started":1}
Aug 06 04:31:59 dist-test-slave-ncp9 krb5kdc[12545](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:31:59.727210 14281 heartbeater.cc:464] Registering TS with master...
I20260806 04:31:59.727557 14281 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:31:59.728936 13979 ts_manager.cc:194] Registered new tserver with Master: c8017c373fe34b4f9dd18d56ac553425 (127.28.94.1:40371)
I20260806 04:31:59.730710 13979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:46727
I20260806 04:32:00.028358 14468 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:00.028728 14468 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:32:00.029714 13979 ts_manager.cc:194] Registered new tserver with Master: 407b9f42607740eba783ebc1f7971a53 (127.28.94.2:45331)
I20260806 04:32:00.030586 13979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:47551
Aug 06 04:32:00 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:32:00.300956 14653 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:00.301283 14653 heartbeater.cc:511] Master 127.28.94.62:39379 requested a full tablet report, sending...
I20260806 04:32:00.302143 13979 ts_manager.cc:194] Registered new tserver with Master: 12fe8dc3ea004aa4bc637e77105d5494 (127.28.94.3:40603)
I20260806 04:32:00.302995 13979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:42557
I20260806 04:32:00.309394 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:32:00.328478 14681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.311226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.28.94.62:39379)
0806 04:32:00.311881 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.311886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.312231 (+ 345us) server_negotiation.cc:407] Connection header received
0806 04:32:00.312273 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.312276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.312325 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.312394 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:00.313315 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.314121 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.315280 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.315660 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.315776 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.324245 (+ 8469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:00.324249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:00.324252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:00.324285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:00.326170 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:00.326678 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:00.326682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:00.326684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:00.326782 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:00.327091 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:00.327094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:00.327096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:00.327329 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:00.327451 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.328141 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.328274 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":85,"threads_started":1}
I20260806 04:32:00.333142 13979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:32:00.338095 13979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 06 04:32:00 dist-test-slave-ncp9 krb5kdc[12545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for HTTP/127.28.94.20@KRBTEST.COM
I20260806 04:32:00.367449 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.356522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40603 (local address 127.0.0.1:52144)
0806 04:32:00.359901 (+ 3379us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.359914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:00.360096 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:00.360759 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:00.360762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:00.360780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:00.360862 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.360869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.362440 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.362444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.364640 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.364651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.366875 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.366878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.366989 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.367032 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:32:00.367152 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:32:00.367275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3318,"mutex_wait_us":194,"thread_start_us":102,"threads_started":1}
I20260806 04:32:00.368069 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.356986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.28.94.3:40603)
0806 04:32:00.359763 (+ 2777us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.359769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.360025 (+ 256us) server_negotiation.cc:407] Connection header received
0806 04:32:00.360251 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.360257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.360465 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.360596 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:00.361018 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.362318 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.364770 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.365827 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.365935 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.366098 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.367813 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.367927 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2670,"thread_start_us":87,"threads_started":1}
I20260806 04:32:00.373842 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.356803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:45331 (local address 127.0.0.1:38792)
0806 04:32:00.362871 (+ 6068us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.363999 (+ 1128us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:00.364087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:00.366646 (+ 2559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:00.366649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:00.366668 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:00.366751 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.366757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.369905 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.369908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.371065 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.371078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.373451 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.373454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.373547 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.373572 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:32:00.373635 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 04:32:00.373706 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6005,"thread_start_us":242,"threads_started":1}
I20260806 04:32:00.374576 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.357261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.28.94.2:45331)
0806 04:32:00.364350 (+ 7089us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.364357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.364381 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:32:00.364461 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.364466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.366312 (+ 1846us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.366468 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:00.368563 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.369761 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.372320 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.373152 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.373253 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.373393 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.374329 (+ 936us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.374435 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6971,"thread_start_us":71,"threads_started":1}
I20260806 04:32:00.374917 14536 tablet_service.cc:1467] Processing CreateTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c (DEFAULT_TABLE table=test-table [id=9e795b405d5e4df4bd17c7d124e836dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:00.376065 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.353810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:40371 (local address 127.0.0.1:35180)
0806 04:32:00.359282 (+ 5472us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.359324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:00.359466 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:00.360236 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:00.360242 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:00.360265 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:00.360398 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.360405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.362656 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.362659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.373644 (+ 10985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.373653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.375666 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.375670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.375761 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.375788 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:32:00.375850 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 04:32:00.375930 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5263,"thread_start_us":119,"threads_started":1}
I20260806 04:32:00.376480 14536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6500ed9b333b4d87bdeef8b3a9cd388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:00.377821 14380 tablet_service.cc:1467] Processing CreateTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c (DEFAULT_TABLE table=test-table [id=9e795b405d5e4df4bd17c7d124e836dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:00.378925 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6500ed9b333b4d87bdeef8b3a9cd388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:00.388394 14699 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Bootstrap starting.
I20260806 04:32:00.391089 14699 tablet_bootstrap.cc:654] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:00.392375 14699 log.cc:824] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:00.393204 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.354867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.28.94.1:40371)
0806 04:32:00.357427 (+ 2560us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.357436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.359429 (+ 1993us) server_negotiation.cc:407] Connection header received
0806 04:32:00.359632 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.359639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.359870 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.360025 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:00.360537 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.362526 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.373785 (+ 11259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.374884 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.376962 (+ 2078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.377129 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.377188 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.393033 (+ 15845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2403,"thread_start_us":132,"threads_started":1}
I20260806 04:32:00.393469 14698 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Bootstrap starting.
I20260806 04:32:00.396831 14160 tablet_service.cc:1467] Processing CreateTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c (DEFAULT_TABLE table=test-table [id=9e795b405d5e4df4bd17c7d124e836dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:00.398150 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6500ed9b333b4d87bdeef8b3a9cd388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:00.399179 14698 tablet_bootstrap.cc:654] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:00.411096 14698 log.cc:824] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:00.416246 14698 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: No bootstrap required, opened a new log
I20260806 04:32:00.418313 14702 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: Bootstrap starting.
I20260806 04:32:00.418481 14699 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: No bootstrap required, opened a new log
I20260806 04:32:00.418716 14699 ts_tablet_manager.cc:1397] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20260806 04:32:00.421340 14702 tablet_bootstrap.cc:654] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:00.423156 14699 raft_consensus.cc:348] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.423460 14699 raft_consensus.cc:374] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:00.423528 14699 raft_consensus.cc:729] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407b9f42607740eba783ebc1f7971a53, State: Initialized, Role: FOLLOWER
I20260806 04:32:00.424063 14699 consensus_queue.cc:260] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.425660 14699 ts_tablet_manager.cc:1428] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:32:00.426064 14468 heartbeater.cc:503] Master 127.28.94.62:39379 was elected leader, sending a full tablet report...
I20260806 04:32:00.426234 14698 ts_tablet_manager.cc:1397] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20260806 04:32:00.430614 14698 raft_consensus.cc:348] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.430924 14698 raft_consensus.cc:374] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:00.430974 14698 raft_consensus.cc:729] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fe8dc3ea004aa4bc637e77105d5494, State: Initialized, Role: FOLLOWER
I20260806 04:32:00.431511 14698 consensus_queue.cc:260] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.434173 14698 ts_tablet_manager.cc:1428] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260806 04:32:00.434556 14653 heartbeater.cc:503] Master 127.28.94.62:39379 was elected leader, sending a full tablet report...
I20260806 04:32:00.428057 14702 log.cc:824] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:00.445465 14702 tablet_bootstrap.cc:492] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: No bootstrap required, opened a new log
I20260806 04:32:00.445829 14702 ts_tablet_manager.cc:1397] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260806 04:32:00.450700 14702 raft_consensus.cc:348] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.451102 14702 raft_consensus.cc:374] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:00.451220 14702 raft_consensus.cc:729] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8017c373fe34b4f9dd18d56ac553425, State: Initialized, Role: FOLLOWER
I20260806 04:32:00.451849 14702 consensus_queue.cc:260] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.453156 14702 ts_tablet_manager.cc:1428] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:32:00.453588 14281 heartbeater.cc:503] Master 127.28.94.62:39379 was elected leader, sending a full tablet report...
W20260806 04:32:00.483595 14469 tablet.cc:2369] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:32:00.528262 14654 tablet.cc:2369] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:00.553332 14704 raft_consensus.cc:482] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:00.553640 14704 raft_consensus.cc:504] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.555231 14704 leader_election.cc:288] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8017c373fe34b4f9dd18d56ac553425 (127.28.94.1:40371), 12fe8dc3ea004aa4bc637e77105d5494 (127.28.94.3:40603)
I20260806 04:32:00.569635 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.557926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:50933 (local address 127.28.94.3:40603)
0806 04:32:00.558062 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.558067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.561729 (+ 3662us) server_negotiation.cc:407] Connection header received
0806 04:32:00.562150 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.562153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.562228 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.562295 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:00.563254 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.563998 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.568215 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.568841 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.568960 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.569025 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.569386 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.569474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:32:00.570384 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.556652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40603 (local address 127.28.94.2:50933)
0806 04:32:00.561646 (+ 4994us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.561658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:00.562021 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:00.562454 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:00.562457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:00.562474 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:00.563113 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.563126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.564217 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.564220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.565338 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.565349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.569119 (+ 3770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.569123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.569241 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.569265 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:32:00.570084 (+ 819us) client_negotiation.cc:240] Negotiation successful
0806 04:32:00.570174 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4935,"thread_start_us":134,"threads_started":1}
I20260806 04:32:00.571331 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6500ed9b333b4d87bdeef8b3a9cd388c" candidate_uuid: "407b9f42607740eba783ebc1f7971a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fe8dc3ea004aa4bc637e77105d5494" is_pre_election: true
I20260806 04:32:00.571743 14572 raft_consensus.cc:2454] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407b9f42607740eba783ebc1f7971a53 in term 0.
I20260806 04:32:00.572304 14307 leader_election.cc:302] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fe8dc3ea004aa4bc637e77105d5494, 407b9f42607740eba783ebc1f7971a53; no voters:
I20260806 04:32:00.572714 14704 raft_consensus.cc:2781] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:00.572815 14704 raft_consensus.cc:482] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:00.572868 14704 raft_consensus.cc:3037] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:00.574384 14704 raft_consensus.cc:504] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.574823 14704 leader_election.cc:288] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [CANDIDATE]: Term 1 election: Requested vote from peers c8017c373fe34b4f9dd18d56ac553425 (127.28.94.1:40371), 12fe8dc3ea004aa4bc637e77105d5494 (127.28.94.3:40603)
I20260806 04:32:00.575548 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6500ed9b333b4d87bdeef8b3a9cd388c" candidate_uuid: "407b9f42607740eba783ebc1f7971a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fe8dc3ea004aa4bc637e77105d5494"
I20260806 04:32:00.575702 14572 raft_consensus.cc:3037] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:00.577235 14572 raft_consensus.cc:2454] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407b9f42607740eba783ebc1f7971a53 in term 1.
I20260806 04:32:00.577279 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.557737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:34185 (local address 127.28.94.1:40371)
0806 04:32:00.558431 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.558436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.561694 (+ 3258us) server_negotiation.cc:407] Connection header received
0806 04:32:00.561915 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.561920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.561980 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.562032 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:00.562591 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.563362 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.564967 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.565563 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.565639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.565715 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.577019 (+ 11304us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.577147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601}
I20260806 04:32:00.577667 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.555981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:40371 (local address 127.28.94.2:34185)
0806 04:32:00.561592 (+ 5611us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.561609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:00.561785 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:00.562211 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:00.562214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:00.562231 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:00.562425 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.562435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.563541 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.563544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.564810 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:00.564821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.565826 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.565829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:00.565931 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.565957 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:32:00.577416 (+ 11459us) client_negotiation.cc:240] Negotiation successful
0806 04:32:00.577517 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5528,"thread_start_us":259,"threads_started":1}
I20260806 04:32:00.577807 14307 leader_election.cc:302] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12fe8dc3ea004aa4bc637e77105d5494, 407b9f42607740eba783ebc1f7971a53; no voters:
I20260806 04:32:00.578775 14210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6500ed9b333b4d87bdeef8b3a9cd388c" candidate_uuid: "407b9f42607740eba783ebc1f7971a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8017c373fe34b4f9dd18d56ac553425" is_pre_election: true
I20260806 04:32:00.579133 14210 raft_consensus.cc:2454] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 407b9f42607740eba783ebc1f7971a53 in term 0.
I20260806 04:32:00.579485 14209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6500ed9b333b4d87bdeef8b3a9cd388c" candidate_uuid: "407b9f42607740eba783ebc1f7971a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8017c373fe34b4f9dd18d56ac553425"
I20260806 04:32:00.579598 14209 raft_consensus.cc:3037] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:00.581120 14209 raft_consensus.cc:2454] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 407b9f42607740eba783ebc1f7971a53 in term 1.
I20260806 04:32:00.582158 14704 raft_consensus.cc:2781] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:00.582432 14704 raft_consensus.cc:686] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 1 LEADER]: Becoming Leader. State: Replica: 407b9f42607740eba783ebc1f7971a53, State: Running, Role: LEADER
I20260806 04:32:00.582816 14704 consensus_queue.cc:237] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } }
I20260806 04:32:00.586819 13978 catalog_manager.cc:5495] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 407b9f42607740eba783ebc1f7971a53 (127.28.94.2). New cstate: current_term: 1 leader_uuid: "407b9f42607740eba783ebc1f7971a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407b9f42607740eba783ebc1f7971a53" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 45331 } health_report { overall_health: HEALTHY } } }
I20260806 04:32:00.615531 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.610437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.28.94.2:45331)
0806 04:32:00.610556 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:00.610561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:00.610577 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:32:00.610662 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:00.610664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:00.610724 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:00.610826 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:00.611114 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.611932 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.613237 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:00.613578 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:00.613692 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:00.614131 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:00.614733 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:00.614861 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:00.615295 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 04:32:00.615367 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 04:32:00.623425 14572 raft_consensus.cc:1261] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 1 FOLLOWER]: Refusing update from remote peer 407b9f42607740eba783ebc1f7971a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:00.624366 14713 consensus_queue.cc:1035] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12fe8dc3ea004aa4bc637e77105d5494" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:00.632418 14209 raft_consensus.cc:1261] T 6500ed9b333b4d87bdeef8b3a9cd388c P c8017c373fe34b4f9dd18d56ac553425 [term 1 FOLLOWER]: Refusing update from remote peer 407b9f42607740eba783ebc1f7971a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:00.633430 14713 consensus_queue.cc:1035] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8017c373fe34b4f9dd18d56ac553425" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:00.652824 14716 mvcc.cc:205] Tried to move back new op lower bound from 7315417991663607808 to 7315417991511244800. Current Snapshot: MvccSnapshot[applied={T|T < 7315417991663607808}]
I20260806 04:32:00.653411 14721 mvcc.cc:205] Tried to move back new op lower bound from 7315417991663607808 to 7315417991511244800. Current Snapshot: MvccSnapshot[applied={T|T < 7315417991663607808}]
I20260806 04:32:00.661681 14717 mvcc.cc:205] Tried to move back new op lower bound from 7315417991663607808 to 7315417991511244800. Current Snapshot: MvccSnapshot[applied={T|T < 7315417991663607808}]
I20260806 04:32:00.663030 13978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:00.663211 13978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:00.666108 13978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c624c4f8eab439b9e7187e84f85a599: Sending DeleteTablet for 3 replicas of tablet 6500ed9b333b4d87bdeef8b3a9cd388c
I20260806 04:32:00.667516 14160 tablet_service.cc:1514] Processing DeleteTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:00 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35180
I20260806 04:32:00.668085 14536 tablet_service.cc:1514] Processing DeleteTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:00 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:52144
I20260806 04:32:00.668519 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet 6500ed9b333b4d87bdeef8b3a9cd388c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:00 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:38792
I20260806 04:32:00.669982 14730 tablet_replica.cc:330] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: stopping tablet replica
I20260806 04:32:00.670017 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14030
I20260806 04:32:00.673297 14732 tablet_replica.cc:330] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: stopping tablet replica
I20260806 04:32:00.673727 14730 raft_consensus.cc:2229] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:00.674080 14732 raft_consensus.cc:2229] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:32:00.674508 14732 raft_consensus.cc:2258] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:00.674902 14730 raft_consensus.cc:2258] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:00.677459 14730 ts_tablet_manager.cc:1905] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:00.677831 14732 ts_tablet_manager.cc:1905] T 6500ed9b333b4d87bdeef8b3a9cd388c P 407b9f42607740eba783ebc1f7971a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:00.680511 14730 ts_tablet_manager.cc:1918] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:00.680682 14730 log.cc:1196] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/6500ed9b333b4d87bdeef8b3a9cd388c
I20260806 04:32:00.681108 14730 ts_tablet_manager.cc:1939] T 6500ed9b333b4d87bdeef8b3a9cd388c P 12fe8dc3ea004aa4bc637e77105d5494: Deleting consensus metadata
I20260806 04:32:00.682569 13949 catalog_manager.cc:4841] TS 12fe8dc3ea004aa4bc637e77105d5494 (127.28.94.3:40603): tablet 6500ed9b333b4d87bdeef8b3a9cd388c (table test-table [id=9e795b405d5e4df4bd17c7d124e836dd]) successfully deleted
W20260806 04:32:00.684947 13950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:40371 (error 108)
I20260806 04:32:00.685878 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14285
I20260806 04:32:00.702056 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.701643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:40371 (local address 127.0.0.1:35184)
0806 04:32:00.701868 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.701955 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 04:32:00.702459 13950 catalog_manager.cc:4584] TS c8017c373fe34b4f9dd18d56ac553425 (127.28.94.1:40371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6500ed9b333b4d87bdeef8b3a9cd388c: Network error: Client connection negotiation failed: client connection to 127.28.94.1:40371: connect: Connection refused (error 111)
I20260806 04:32:00.710309 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14470
I20260806 04:32:00.710523 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:00.710180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:45331 (local address 127.0.0.1:38812)
0806 04:32:00.710382 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:00.710432 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.2:45331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 04:32:00.724505 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 13930
I20260806 04:32:00.739301 29048 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:32:01.083410 29048 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:32:01.083515 29048 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:32:01.426949 29048 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:32:01.427 UTC [12586] LOG: received smart shutdown request
2026-08-06 04:32:01.431 UTC [12586] LOG: background worker "logical replication launcher" (PID 12600) exited with exit code 1
2026-08-06 04:32:01.431 UTC [12593] LOG: shutting down
2026-08-06 04:32:01.570 UTC [12586] LOG: database system is shut down
2026-08-06T04:32:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48436 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:32:01 dist-test-slave-ncp9 krb5kdc[14753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:32:01 dist-test-slave-ncp9 krb5kdc[14753](info): set up 2 sockets
Aug 06 04:32:01 dist-test-slave-ncp9 krb5kdc[14753](info): commencing operation
krb5kdc: starting...
W20260806 04:32:02.767453 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:32:02 dist-test-slave-ncp9 krb5kdc[14753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:32:02.881929 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46619
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:02.991441 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:02.991701 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:02.991758 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:02.995726 14778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:02.995782 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:02.995807 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:02.995827 14778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:02.995851 14778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:03.000933 14778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:03.001920 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:03.002900 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:03.010792 14784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:03.010841 14786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:03.010859 14783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:03.011188 14778 server_base.cc:1034] running on GCE node
I20260806 04:32:03.011716 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:03.012693 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:03.013861 14778 hybrid_clock.cc:648] HybridClock initialized: now 1785990723013843 us; error 27 us; skew 500 ppm
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:03.016402 14778 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:03.017241 14778 webserver.cc:460] Webserver started at http://127.28.94.62:40919/ using document root <none> and password file <none>
I20260806 04:32:03.017728 14778 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:03.017784 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:03.017946 14778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:03.019797 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "2e9d984cb2c1cfeba61ae9f2238af4a4"
server_key_iv: "8cc9860b61694109797148714b2c259d"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.020282 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "2e9d984cb2c1cfeba61ae9f2238af4a4"
server_key_iv: "8cc9860b61694109797148714b2c259d"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.023676 14778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:32:03.025897 14795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:03.027040 14778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:32:03.027153 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "2e9d984cb2c1cfeba61ae9f2238af4a4"
server_key_iv: "8cc9860b61694109797148714b2c259d"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.027240 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:03.045985 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:03.048445 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:03.048606 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:03.055300 14778 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46619
I20260806 04:32:03.055308 14864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46619 every 8 connection(s)
I20260806 04:32:03.056188 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:03.056982 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14778
I20260806 04:32:03.057091 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:32:03.057353 29048 external_mini_cluster.cc:1428] Setting key 04b7b26698ebe5c18c30c3d809a0de8e
I20260806 04:32:03.058923 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:03.063741 14865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.61@KRBTEST.COM" created.
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:03.078130 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.066357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.28.94.62:46619)
0806 04:32:03.066886 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.066902 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.067050 (+ 148us) server_negotiation.cc:407] Connection header received
0806 04:32:03.068014 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.068019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.068197 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.068343 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.069060 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.069987 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.070977 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.071333 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.071453 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.073703 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.073710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.073723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.073749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.075452 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.075880 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.075887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.075889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.076032 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.076305 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.076308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.076310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.076782 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.076949 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.077262 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.077488 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":412,"spinlock_wait_cycles":31104,"thread_start_us":153,"threads_started":1}
I20260806 04:32:03.078138 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.066357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46619 (local address 127.0.0.1:52138)
0806 04:32:03.066892 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.066944 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.067744 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.068353 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.068357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.068650 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.068927 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.068946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.070099 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.070103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.070847 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.070859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.071482 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.071487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.071621 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.072188 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.072203 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.073558 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.075550 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.075557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.075564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.075775 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.076123 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.076125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.076126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.076222 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.076881 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.076885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.077026 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.077165 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.077393 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260806 04:32:03.080205 14865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:32:03.080852 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.080575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35474)
0806 04:32:03.080682 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.080765 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 04:32:03.081051 14796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
I20260806 04:32:03.081806 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.081618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35478)
0806 04:32:03.081717 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.081759 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:32:03.081923 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41159: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
I20260806 04:32:03.083637 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:41159
--webserver_interface=127.28.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:03.131174 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } attempt: 1
I20260806 04:32:03.131938 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.131671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35486)
0806 04:32:03.131806 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.131864 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:32:03.132582 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.132382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35490)
0806 04:32:03.132481 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.132523 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 04:32:03.132735 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41159: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
W20260806 04:32:03.192107 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:03.192332 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:03.192384 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:03.196305 14873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:03.196360 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:03.196386 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:03.196408 14873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:03.196436 14873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:03.200876 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } attempt: 2
I20260806 04:32:03.201651 14873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:03.201725 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.201418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35502)
0806 04:32:03.201572 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.201645 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 04:32:03.202459 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.202240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35516)
0806 04:32:03.202355 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.202400 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:32:03.202580 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41159: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41159: connect: Connection refused (error 111)
I20260806 04:32:03.202693 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:03.203651 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:03.211647 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:03.211658 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:03.211669 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:03.211838 14873 server_base.cc:1034] running on GCE node
I20260806 04:32:03.212560 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:03.213598 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:03.214747 14873 hybrid_clock.cc:648] HybridClock initialized: now 1785990723214728 us; error 28 us; skew 500 ppm
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:03.217291 14873 init.cc:377] Logged in from keytab as kudu/127.28.94.61@KRBTEST.COM (short username kudu)
I20260806 04:32:03.218112 14873 webserver.cc:460] Webserver started at http://127.28.94.61:36103/ using document root <none> and password file <none>
I20260806 04:32:03.218631 14873 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:03.218698 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:03.218902 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:03.220801 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data/instance:
uuid: "7cf59fa1f758421c9d11eec43b59fb83"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "10255dcdb5ce1f3d3e4fe0acc69c4dca"
server_key_iv: "aae0112a23657a02c934f7d40721f5a8"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.221374 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal/instance:
uuid: "7cf59fa1f758421c9d11eec43b59fb83"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "10255dcdb5ce1f3d3e4fe0acc69c4dca"
server_key_iv: "aae0112a23657a02c934f7d40721f5a8"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.224773 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:03.227016 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:03.228099 14873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:03.228209 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "7cf59fa1f758421c9d11eec43b59fb83"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "10255dcdb5ce1f3d3e4fe0acc69c4dca"
server_key_iv: "aae0112a23657a02c934f7d40721f5a8"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.228294 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:03.266366 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:03.269038 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:03.269210 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:03.275830 14873 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.61:41159
I20260806 04:32:03.275869 14978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.61:41159 every 8 connection(s)
I20260806 04:32:03.276755 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
I20260806 04:32:03.279361 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14873
I20260806 04:32:03.279438 14979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:03.279460 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-1/wal/instance
I20260806 04:32:03.279712 29048 external_mini_cluster.cc:1428] Setting key 3a0f77e79fe435171465ca86ecb667e0
I20260806 04:32:03.283165 14979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.94.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.60@KRBTEST.COM" created.
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:03.293751 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } attempt: 3
I20260806 04:32:03.296579 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.284940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.28.94.62:46619)
0806 04:32:03.285076 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.285080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.285947 (+ 867us) server_negotiation.cc:407] Connection header received
0806 04:32:03.287050 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.287052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.287099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.287169 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.288626 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.289209 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.290306 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.290535 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.290618 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.292827 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.292830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.292832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.292856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.294328 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.294914 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.294918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.294920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.295021 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.295378 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.295382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.295384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.295577 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.295676 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.296303 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.296419 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:32:03.297556 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.285268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46619 (local address 127.0.0.1:52142)
0806 04:32:03.285770 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.285828 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.286844 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.287370 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.287380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.287815 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.288468 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.288491 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.289326 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.289329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.290198 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.290208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.290645 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.290648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.290778 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.291231 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.291246 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.292669 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.294433 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.294439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.294452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.294806 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.295120 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.295123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.295125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.295293 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.295690 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.295697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.296003 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.296225 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.296539 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260806 04:32:03.299217 14979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:32:03.303900 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.294167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35526)
0806 04:32:03.294508 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.294522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.294590 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.295197 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.295200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.295356 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.295470 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.295477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.296636 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.296640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.297263 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.297272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.297932 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.297936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.298055 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.298499 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.298514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.300395 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.302285 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.302288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.302290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.302481 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.302749 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.302752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.302754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.302968 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.303472 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.303475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.303572 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.303639 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.303735 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:32:03.304803 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.294376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.28.94.61:41159)
0806 04:32:03.294807 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.294818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.294840 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:32:03.294879 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.294884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.295042 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.295224 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.295588 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.296482 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.297381 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.297780 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.297918 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.300546 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.300556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.300568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.300594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.302180 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.302580 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.302587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.302588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.302663 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.303059 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.303062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.303064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.303348 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.304136 (+ 788us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.304247 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.304378 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":99,"threads_started":1}
I20260806 04:32:03.306481 14865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:32:03.307147 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.306901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36716)
0806 04:32:03.307046 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.307084 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
I20260806 04:32:03.307834 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.300332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35528)
0806 04:32:03.300498 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.300511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.300584 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.300863 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.300865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.301043 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.301132 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.301138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.301840 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.301844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.302479 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.302491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.302900 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.302904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.303057 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.303421 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.303431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.304732 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.306488 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.306491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.306493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.306716 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.306977 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.306980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.306981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.307057 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.307391 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.307394 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.307474 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.307581 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.307697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":43}
I20260806 04:32:03.307890 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.307611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36730)
0806 04:32:03.307792 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.307839 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 04:32:03.307977 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.300401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.28.94.61:41159)
0806 04:32:03.300691 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.300699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.300711 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:32:03.300735 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.300737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.300767 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.300836 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.301229 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.301747 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.302586 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.302810 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.302892 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.304881 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.304961 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.304963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.304989 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.306404 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.306805 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.306809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.306810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.306887 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.307131 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.307134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.307136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.307308 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.307391 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.307682 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.307798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
W20260806 04:32:03.308120 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
I20260806 04:32:03.308301 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:35877
--webserver_interface=127.28.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:03.308496 14979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:32:03.309059 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.308825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36742)
0806 04:32:03.308941 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.309011 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 04:32:03.309219 14896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
I20260806 04:32:03.309918 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.309739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36754)
0806 04:32:03.309829 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.309873 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 04:32:03.310030 14979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
I20260806 04:32:03.339259 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 1
I20260806 04:32:03.339915 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.339673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36764)
0806 04:32:03.339796 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.339849 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 04:32:03.340541 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.340346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36768)
0806 04:32:03.340447 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.340488 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:32:03.340677 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
I20260806 04:32:03.359244 14979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 1
I20260806 04:32:03.359937 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.359696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36782)
0806 04:32:03.359822 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.359873 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 04:32:03.360577 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.360368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36786)
0806 04:32:03.360466 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.360521 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 04:32:03.360680 14979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
I20260806 04:32:03.415818 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 2
I20260806 04:32:03.416630 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.416336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36794)
0806 04:32:03.416483 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.416544 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:32:03.417332 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.417121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36802)
0806 04:32:03.417230 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.417277 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 04:32:03.417482 14865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
W20260806 04:32:03.417797 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:03.418016 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:03.418079 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:03.421981 14987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:03.422030 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:03.422055 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:03.422086 14987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:03.422113 14987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:03.427124 14987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46619,127.28.94.61:41159,127.28.94.60:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:03.428054 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:03.428815 14979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 2
I20260806 04:32:03.429031 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:32:03.429477 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.429241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36812)
0806 04:32:03.429360 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.429411 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 04:32:03.430155 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.429922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36824)
0806 04:32:03.430023 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.430085 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:32:03.430270 14979 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:35877: Network error: Client connection negotiation failed: client connection to 127.28.94.60:35877: connect: Connection refused (error 111)
W20260806 04:32:03.437100 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:03.437098 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:03.437222 14987 server_base.cc:1034] running on GCE node
W20260806 04:32:03.437105 15001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:03.437911 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:03.438921 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:03.440081 14987 hybrid_clock.cc:648] HybridClock initialized: now 1785990723440065 us; error 24 us; skew 500 ppm
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:03.442649 14987 init.cc:377] Logged in from keytab as kudu/127.28.94.60@KRBTEST.COM (short username kudu)
I20260806 04:32:03.443585 14987 webserver.cc:460] Webserver started at http://127.28.94.60:39389/ using document root <none> and password file <none>
I20260806 04:32:03.444116 14987 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:03.444180 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:03.444366 14987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:03.446338 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data/instance:
uuid: "fb374251ca7147bbbbd4c0613607e425"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "0815ae4a6d6c8685654e5fbb9addcd12"
server_key_iv: "04ed3d0ca2ebf344e891fff9aa960104"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.446883 14987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal/instance:
uuid: "fb374251ca7147bbbbd4c0613607e425"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "0815ae4a6d6c8685654e5fbb9addcd12"
server_key_iv: "04ed3d0ca2ebf344e891fff9aa960104"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.451463 14987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:32:03.453907 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:03.455021 14987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:32:03.455138 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "fb374251ca7147bbbbd4c0613607e425"
format_stamp: "Formatted at 2026-08-06 04:32:03 on dist-test-slave-ncp9"
server_key: "0815ae4a6d6c8685654e5fbb9addcd12"
server_key_iv: "04ed3d0ca2ebf344e891fff9aa960104"
server_key_version: "encryptionkey@0"
I20260806 04:32:03.455240 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:03.470877 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:03.473327 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:03.473487 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:03.479809 14987 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.60:35877
I20260806 04:32:03.479861 15086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.60:35877 every 8 connection(s)
I20260806 04:32:03.480688 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
I20260806 04:32:03.483336 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:03.483948 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 14987
I20260806 04:32:03.484063 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785990554567992-29048-0/minicluster-data/master-2/wal/instance
I20260806 04:32:03.484359 29048 external_mini_cluster.cc:1428] Setting key 223f84604746acaf4f647591b0f7e738
I20260806 04:32:03.488207 15087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:03.498458 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.485593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.28.94.60:35877)
0806 04:32:03.486009 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.486026 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.486069 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:32:03.486758 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.486783 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.487183 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.487520 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.488133 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.489558 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.490716 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.491174 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.491316 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.493784 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.493791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.493802 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.493835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.495596 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.496019 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.496024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.496028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.496172 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.496446 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.496449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.496450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.496887 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.497072 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.497344 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.497604 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
I20260806 04:32:03.499577 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.490066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46619 (local address 127.0.0.1:52152)
0806 04:32:03.490433 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.490470 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.490579 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.490858 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.490863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.491182 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.491301 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.491307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.492148 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.492151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.492860 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.492872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.493352 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.493357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.493471 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.493981 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.493993 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.495686 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.497425 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.497433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.497438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.497736 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.497984 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.497986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.497988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.498078 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.498408 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.498413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.498531 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.499332 (+ 801us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.499461 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":158,"threads_started":1}
I20260806 04:32:03.499676 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.490114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.28.94.62:46619)
0806 04:32:03.490229 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.490232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.490547 (+ 315us) server_negotiation.cc:407] Connection header received
0806 04:32:03.490682 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.490685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.490734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.490801 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.491419 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.492034 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.492994 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.493208 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.493314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.495811 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.495815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.495817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.495847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.497328 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.497828 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.497831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.497832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.497904 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.498155 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.498157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.498158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.498319 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.498386 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.499411 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.499529 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:32:03.500658 15087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:32:03.508102 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.500566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.28.94.62:46619)
0806 04:32:03.500666 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.500669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.500683 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:32:03.500783 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.500786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.500826 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.500925 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.501556 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.501996 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.502798 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.503050 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.503129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.505323 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.505326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.505328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.505350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.506639 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507016 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.507020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.507022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.507099 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507350 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.507353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.507355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.507544 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.507745 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.507853 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.507961 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:32:03.508471 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.501101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41159 (local address 127.0.0.1:35534)
0806 04:32:03.501610 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.501620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.501694 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.501901 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.501903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.502049 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.502152 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.502159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.502856 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.502859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.503491 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.503502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.503948 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.503953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.504057 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.504398 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.504412 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.505502 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.507008 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.507013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.507220 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.507489 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.507492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.507565 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.508007 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.508010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.508102 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.508253 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.508352 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449}
I20260806 04:32:03.508594 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.501251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.28.94.61:41159)
0806 04:32:03.501418 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.501422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.501682 (+ 260us) server_negotiation.cc:407] Connection header received
0806 04:32:03.501767 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.501770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.501814 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.501880 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.502269 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.502752 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.503609 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.503837 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.503914 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.505619 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.505623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.505625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.505654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.506905 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507310 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.507313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.507315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.507402 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.507654 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.507657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.507658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.507880 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.507975 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.508333 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.508449 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:32:03.509246 15087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:32:03.516324 15094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.509656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36840)
0806 04:32:03.509813 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.509823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.509886 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.510070 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.510072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.510192 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.510255 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.510259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.511239 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.511242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.511725 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.511733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.512141 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.512145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.512214 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.512569 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.512579 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.513577 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.515027 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.515030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.515032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.515215 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.515446 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.515449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.515451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.515519 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.515843 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.515846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.515926 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.516030 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.516148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 04:32:03.516369 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.509694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.28.94.60:35877)
0806 04:32:03.509795 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.509799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.509861 (+ 62us) server_negotiation.cc:407] Connection header received
0806 04:32:03.509941 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.509944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.509984 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.510063 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.510355 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.511139 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.511824 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.512045 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.512124 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.513696 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.513700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.513702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.513725 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.514932 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.515297 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.515300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.515302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.515370 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.515596 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.515599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.515601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.515757 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.515838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.516101 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.516202 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 04:32:03.516631 14865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 3
I20260806 04:32:03.517812 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.509258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.28.94.61:41159)
0806 04:32:03.509378 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.509382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.509442 (+ 60us) server_negotiation.cc:407] Connection header received
0806 04:32:03.509534 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.509537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.509572 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.509643 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.510299 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.510746 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.511426 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.511642 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.511723 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.514262 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.514265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.514267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.514288 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.515508 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.516208 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.516212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.516213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.516296 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.516832 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.516835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.516837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.517197 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.517388 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.517542 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.517666 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:32:03.519701 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: Bootstrap starting.
I20260806 04:32:03.521159 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:32:03.521410 14979 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } attempt: 3
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:32:03.522545 15087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:03.523447 15087 log.cc:824] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:03.525501 15087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: No bootstrap required, opened a new log
I20260806 04:32:03.526217 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.517001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36854)
0806 04:32:03.517117 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.517140 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.517210 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.517622 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.517625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.517783 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.517877 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.517884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.518803 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.518806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.519418 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.519430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.520237 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.520241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.520324 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.520726 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.520739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.522271 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.524294 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.524298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.524300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.524533 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.524836 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.524839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.524841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.524959 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.525736 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.525740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.525840 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.525923 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.526052 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
Aug 06 04:32:03 dist-test-slave-ncp9 krb5kdc[14753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:32:03.527050 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.517095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.28.94.60:35877)
0806 04:32:03.517180 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.517184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.517196 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:32:03.517458 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.517462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.517504 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.518085 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.518181 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.518703 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.519547 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.520011 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.520157 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.522400 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.522404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.522406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.522431 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.524173 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.524645 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.524648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.524650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.524728 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.525077 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.525080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.525082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.525328 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.526672 (+ 1344us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.526751 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.526851 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 04:32:03.530154 14865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: Bootstrap starting.
I20260806 04:32:03.530821 14982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.521749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:35877 (local address 127.0.0.1:36858)
0806 04:32:03.521845 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:03.521859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:03.521927 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:03.522747 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:03.522750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:03.522918 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:03.523008 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.523013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.523960 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.523963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.524729 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:03.524741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.525422 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.525426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:03.525508 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.525922 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:03.525937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:03.527315 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:03.529294 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.529297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.529299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.529524 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.529836 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:03.529840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:03.529841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:03.529930 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:03.530376 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:03.530379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:03.530468 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 04:32:03.530562 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 04:32:03.530685 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 04:32:03.530903 15087 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.531152 15087 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:03.531209 15087 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb374251ca7147bbbbd4c0613607e425, State: Initialized, Role: FOLLOWER
I20260806 04:32:03.531380 15096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.521956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.28.94.60:35877)
0806 04:32:03.522538 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.522548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.522561 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:32:03.522592 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.522594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.522646 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.522725 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.523118 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.523826 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.524845 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.525220 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.525315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.527463 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.527467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.527469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.527494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.529175 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.529634 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.529637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.529639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.529734 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.530029 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.530032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.530034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.530242 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.530330 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.530997 (+ 667us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.531176 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":127,"threads_started":1}
I20260806 04:32:03.531759 15087 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.532788 14865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:03.533277 15087 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:03.533658 14865 log.cc:824] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:03.534168 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: Bootstrap starting.
I20260806 04:32:03.534191 15111 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.534296 15111 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.536075 14865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: No bootstrap required, opened a new log
I20260806 04:32:03.536721 14979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:03.537423 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.525100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.28.94.62:46619)
0806 04:32:03.525212 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.525216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.527856 (+ 2640us) server_negotiation.cc:407] Connection header received
0806 04:32:03.527949 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.527952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.527991 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.528052 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.528784 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.529512 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.530716 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.531026 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.531101 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.533545 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.533548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.533550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.533576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.535321 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.535993 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.535997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.535998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.536085 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.536441 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.536444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.536445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.536653 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.536759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.537173 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.537284 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:32:03.537529 14979 log.cc:824] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:03.539414 14865 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.539628 14865 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:03.539700 14865 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b6a30d1a2a4ac7b03bcddc51634bed, State: Initialized, Role: FOLLOWER
I20260806 04:32:03.540120 14865 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.540524 15129 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.540652 15129 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.541469 14865 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:03.541659 14987 master_runner.cc:186] Error getting master registration for 127.28.94.62:46619: OK, instance_id { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" instance_seqno: 1785990723053431 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:32:03.545534 14987 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 04:32:03.547762 15142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:32:03.547897 15142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:32:03.547925 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.526220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.28.94.60:35877)
0806 04:32:03.527374 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.527378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.537749 (+ 10371us) server_negotiation.cc:407] Connection header received
0806 04:32:03.537790 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.537793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.537834 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.537902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.539735 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.540758 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.542268 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.542591 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.542653 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.544014 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.544018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.544020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.544045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.545530 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.546326 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.546329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.546331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.546416 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.546708 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.546712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.546713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.546911 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.547071 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.547669 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.547775 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":86,"threads_started":1}
I20260806 04:32:03.548768 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:03.525661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.28.94.61:41159)
0806 04:32:03.525757 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:03.525761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:03.535568 (+ 9807us) server_negotiation.cc:407] Connection header received
0806 04:32:03.536203 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:03.536206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:03.536252 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:03.536299 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:03.537605 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.538282 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.539319 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:03.539616 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:03.539734 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:03.541295 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:03.541299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:03.541301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:03.541326 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:03.546745 (+ 5419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.547251 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.547255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.547257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.547341 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:03.547613 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:03.547616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:03.547618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:03.547811 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:03.547919 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:03.548507 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 04:32:03.548624 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:32:03.545915 14778 master_runner.cc:186] Error getting master registration for 127.28.94.61:41159: OK, instance_id { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" instance_seqno: 1785990723274038 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:32:03.549224 14778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 04:32:03.550027 14803 tablet.cc:2369] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:03.550697 14979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: No bootstrap required, opened a new log
W20260806 04:32:03.552181 15125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:32:03.552248 15125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:32:03.554203 14979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.554368 14979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:03.554416 14979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf59fa1f758421c9d11eec43b59fb83, State: Initialized, Role: FOLLOWER
I20260806 04:32:03.554822 14979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.555814 14979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:03.563264 14873 master_runner.cc:263] No leader master found from master 7cf59fa1f758421c9d11eec43b59fb83
I20260806 04:32:03.563321 14873 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:32:03.565068 15146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.565191 15146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 04:32:03.567711 15163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:32:03.567791 15163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:32:03.672436 15111 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:03.672703 15111 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.673923 15111 leader_election.cc:288] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6b6a30d1a2a4ac7b03bcddc51634bed (127.28.94.62:46619), 7cf59fa1f758421c9d11eec43b59fb83 (127.28.94.61:41159)
I20260806 04:32:03.674729 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb374251ca7147bbbbd4c0613607e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf59fa1f758421c9d11eec43b59fb83" is_pre_election: true
I20260806 04:32:03.675155 14937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb374251ca7147bbbbd4c0613607e425 in term 0.
I20260806 04:32:03.675911 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb374251ca7147bbbbd4c0613607e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" is_pre_election: true
I20260806 04:32:03.676249 14831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb374251ca7147bbbbd4c0613607e425 in term 0.
I20260806 04:32:03.676872 15011 leader_election.cc:302] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b6a30d1a2a4ac7b03bcddc51634bed, fb374251ca7147bbbbd4c0613607e425; no voters:
I20260806 04:32:03.677264 15111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:03.677366 15111 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:03.677419 15111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:03.678835 15111 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.679250 15111 leader_election.cc:288] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [CANDIDATE]: Term 1 election: Requested vote from peers a6b6a30d1a2a4ac7b03bcddc51634bed (127.28.94.62:46619), 7cf59fa1f758421c9d11eec43b59fb83 (127.28.94.61:41159)
I20260806 04:32:03.680027 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb374251ca7147bbbbd4c0613607e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed"
I20260806 04:32:03.680177 14831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:03.681649 14831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb374251ca7147bbbbd4c0613607e425 in term 1.
I20260806 04:32:03.682379 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb374251ca7147bbbbd4c0613607e425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf59fa1f758421c9d11eec43b59fb83"
I20260806 04:32:03.682543 14937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:03.683926 14937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb374251ca7147bbbbd4c0613607e425 in term 1.
I20260806 04:32:03.684360 15011 leader_election.cc:302] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf59fa1f758421c9d11eec43b59fb83, fb374251ca7147bbbbd4c0613607e425; no voters:
I20260806 04:32:03.684706 15111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:03.685003 15111 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [term 1 LEADER]: Becoming Leader. State: Replica: fb374251ca7147bbbbd4c0613607e425, State: Running, Role: LEADER
I20260806 04:32:03.685429 15111 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } }
I20260806 04:32:03.687541 15111 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb374251ca7147bbbbd4c0613607e425. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.687711 15111 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:03.694869 15168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:03.695888 15168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:03.718840 14831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [term 1 FOLLOWER]: Refusing update from remote peer fb374251ca7147bbbbd4c0613607e425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:03.720108 14937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [term 1 FOLLOWER]: Refusing update from remote peer fb374251ca7147bbbbd4c0613607e425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:03.720880 15165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:03.721143 15111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:32:03.724722 15016 tablet.cc:2369] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:03.725891 15129 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb374251ca7147bbbbd4c0613607e425. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.726040 15129 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.739952 15146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb374251ca7147bbbbd4c0613607e425. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.740141 15146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.741899 15165 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.742075 15165 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:03.742826 15129 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.742961 15129 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.744338 15165 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.744454 15165 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:03.746567 15168 catalog_manager.cc:1347] Generated new cluster ID: d32cd7c993ad492d84034d25d6338e92
I20260806 04:32:03.746637 15168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:03.751993 15146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb374251ca7147bbbbd4c0613607e425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b6a30d1a2a4ac7b03bcddc51634bed" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46619 } } peers { permanent_uuid: "7cf59fa1f758421c9d11eec43b59fb83" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41159 } } peers { permanent_uuid: "fb374251ca7147bbbbd4c0613607e425" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 35877 } } }
I20260806 04:32:03.752168 15146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:32:03.784561 15168 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:32:03.785972 15168 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:03.801339 15168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb374251ca7147bbbbd4c0613607e425: Generated new TSK 0
I20260806 04:32:03.802474 15168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:32:04.549486 15142 catalog_manager.cc:1259] Loaded cluster ID: d32cd7c993ad492d84034d25d6338e92
I20260806 04:32:04.549571 15142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: loading cluster ID for follower catalog manager: success
I20260806 04:32:04.551254 15142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: acquiring CA information for follower catalog manager: success
I20260806 04:32:04.552446 15142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6b6a30d1a2a4ac7b03bcddc51634bed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:32:04.569183 15163 catalog_manager.cc:1259] Loaded cluster ID: d32cd7c993ad492d84034d25d6338e92
I20260806 04:32:04.569269 15163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: loading cluster ID for follower catalog manager: success
I20260806 04:32:04.570962 15163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: acquiring CA information for follower catalog manager: success
I20260806 04:32:04.572182 15163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7cf59fa1f758421c9d11eec43b59fb83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:32:04.873737 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14778
I20260806 04:32:04.886026 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14873
I20260806 04:32:04.899283 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 14987
2026-08-06T04:32:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3229 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:32:04 dist-test-slave-ncp9 krb5kdc[15189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:32:04 dist-test-slave-ncp9 krb5kdc[15189](info): set up 2 sockets
Aug 06 04:32:04 dist-test-slave-ncp9 krb5kdc[15189](info): commencing operation
krb5kdc: starting...
W20260806 04:32:06.016762 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:32:06.156430 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:41135
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:06.280799 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:06.281221 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:06.281289 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:06.285633 15205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:06.285722 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:06.285749 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:06.285772 15205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:06.285791 15205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:06.291639 15205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:06.292932 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:06.294061 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:06.303303 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:06.303303 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:06.303457 15205 server_base.cc:1034] running on GCE node
W20260806 04:32:06.303671 15221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:06.304347 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:06.305656 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:06.306828 15205 hybrid_clock.cc:648] HybridClock initialized: now 1785990726306813 us; error 35 us; skew 500 ppm
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990726, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:06.309927 15205 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:06.311018 15205 webserver.cc:460] Webserver started at http://127.28.94.62:38161/ using document root <none> and password file <none>
I20260806 04:32:06.311655 15205 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:06.311728 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:06.311965 15205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:06.314384 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "0c5556f0473d4b87bd48361f25457a7d"
format_stamp: "Formatted at 2026-08-06 04:32:06 on dist-test-slave-ncp9"
server_key: "7dcd71294ef68e00f94d0ebf3cd02067"
server_key_iv: "49c679ce1c4f9a8065ac751088eaa3c7"
server_key_version: "encryptionkey@0"
I20260806 04:32:06.315022 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "0c5556f0473d4b87bd48361f25457a7d"
format_stamp: "Formatted at 2026-08-06 04:32:06 on dist-test-slave-ncp9"
server_key: "7dcd71294ef68e00f94d0ebf3cd02067"
server_key_iv: "49c679ce1c4f9a8065ac751088eaa3c7"
server_key_version: "encryptionkey@0"
I20260806 04:32:06.319711 15205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:32:06.322642 15233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:06.324072 15205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:32:06.324257 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "0c5556f0473d4b87bd48361f25457a7d"
format_stamp: "Formatted at 2026-08-06 04:32:06 on dist-test-slave-ncp9"
server_key: "7dcd71294ef68e00f94d0ebf3cd02067"
server_key_iv: "49c679ce1c4f9a8065ac751088eaa3c7"
server_key_version: "encryptionkey@0"
I20260806 04:32:06.324393 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:06.349413 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:06.353029 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:06.353274 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:06.361095 15205 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:41135
I20260806 04:32:06.361128 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:41135 every 8 connection(s)
I20260806 04:32:06.362186 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:06.365545 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 15205
I20260806 04:32:06.365573 15312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:06.365681 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:32:06.366010 29048 external_mini_cluster.cc:1428] Setting key 57e75b0364dca42ad367249516fa0a4d
I20260806 04:32:06.373009 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d: Bootstrap starting.
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:06.375746 15312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:06.376729 15312 log.cc:824] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:06.379323 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d: No bootstrap required, opened a new log
I20260806 04:32:06.381472 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:06.367641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.28.94.62:41135)
0806 04:32:06.368197 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:06.368214 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:06.368256 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:32:06.369284 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:06.369310 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:06.369722 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:06.370066 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:06.370944 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:06.371987 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:06.372876 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:06.373466 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:06.373640 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:06.376433 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:06.376440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:06.376457 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:06.376493 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:06.378319 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:06.378844 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:06.378849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:06.378854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:06.378967 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:06.379277 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:06.379279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:06.379280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:06.379776 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:06.379930 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:06.380261 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 04:32:06.380570 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":180,"threads_started":1}
I20260806 04:32:06.383471 15312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } }
I20260806 04:32:06.383809 15312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:06.383881 15312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c5556f0473d4b87bd48361f25457a7d, State: Initialized, Role: FOLLOWER
I20260806 04:32:06.384385 15312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } }
I20260806 04:32:06.384552 15312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:06.384634 15312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:06.384753 15312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:06.384800 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:32:06.385875 15312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } }
I20260806 04:32:06.386253 15312 leader_election.cc:302] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c5556f0473d4b87bd48361f25457a7d; no voters:
I20260806 04:32:06.386629 15312 leader_election.cc:288] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:32:06.386730 15318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:06.386920 15318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [term 1 LEADER]: Becoming Leader. State: Replica: 0c5556f0473d4b87bd48361f25457a7d, State: Running, Role: LEADER
I20260806 04:32:06.387262 15318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } }
I20260806 04:32:06.388144 15312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:06.389250 15320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c5556f0473d4b87bd48361f25457a7d. Latest consensus state: current_term: 1 leader_uuid: "0c5556f0473d4b87bd48361f25457a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } } }
I20260806 04:32:06.389390 15320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:06.389708 15321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c5556f0473d4b87bd48361f25457a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5556f0473d4b87bd48361f25457a7d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 41135 } } }
I20260806 04:32:06.389837 15330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:06.389853 15321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:06.392951 15330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:06.399578 15330 catalog_manager.cc:1347] Generated new cluster ID: 42a271ac4daf4da3a29d3400847a09a9
I20260806 04:32:06.399685 15330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:06.416051 15330 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:32:06.417546 15330 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:06.432813 15330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c5556f0473d4b87bd48361f25457a7d: Generated new TSK 0
I20260806 04:32:06.433923 15330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:32:06.440979 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 15205
2026-08-06T04:32:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1546 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15347](info): set up 2 sockets
Aug 06 04:32:06 dist-test-slave-ncp9 krb5kdc[15347](info): commencing operation
krb5kdc: starting...
W20260806 04:32:07.552990 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:32:07 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:32:07.672591 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:07.785213 15366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:07.785456 15366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:07.785494 15366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:07.789710 15366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:07.789765 15366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:07.789783 15366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:07.789798 15366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:07.789819 15366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:07.794910 15366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:07.795863 15366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:07.796824 15366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:07.805032 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:07.805054 15366 server_base.cc:1034] running on GCE node
W20260806 04:32:07.804996 15378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:07.804996 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:07.805734 15366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:07.806725 15366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:07.807878 15366 hybrid_clock.cc:648] HybridClock initialized: now 1785990727807863 us; error 25 us; skew 500 ppm
Aug 06 04:32:07 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:07.810411 15366 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:07.811231 15366 webserver.cc:460] Webserver started at http://127.28.94.62:39717/ using document root <none> and password file <none>
I20260806 04:32:07.811744 15366 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:07.811803 15366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:07.812000 15366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:07.813997 15366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "fd301212a9a84edf8f8528ec773394d4"
format_stamp: "Formatted at 2026-08-06 04:32:07 on dist-test-slave-ncp9"
server_key: "392a55c5ef47ed0444ad130724a20926"
server_key_iv: "40fd3066abd95bd4d167405fd939cabc"
server_key_version: "encryptionkey@0"
I20260806 04:32:07.814529 15366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "fd301212a9a84edf8f8528ec773394d4"
format_stamp: "Formatted at 2026-08-06 04:32:07 on dist-test-slave-ncp9"
server_key: "392a55c5ef47ed0444ad130724a20926"
server_key_iv: "40fd3066abd95bd4d167405fd939cabc"
server_key_version: "encryptionkey@0"
I20260806 04:32:07.818051 15366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:07.820272 15388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:07.821444 15366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:32:07.821556 15366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "fd301212a9a84edf8f8528ec773394d4"
format_stamp: "Formatted at 2026-08-06 04:32:07 on dist-test-slave-ncp9"
server_key: "392a55c5ef47ed0444ad130724a20926"
server_key_iv: "40fd3066abd95bd4d167405fd939cabc"
server_key_version: "encryptionkey@0"
I20260806 04:32:07.821643 15366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:07.842103 15366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:07.844741 15366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:07.844954 15366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:07.851725 15366 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46715
I20260806 04:32:07.851956 15464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46715 every 8 connection(s)
I20260806 04:32:07.852681 15366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:07.855422 15470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:07.858083 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 15366
I20260806 04:32:07.858199 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:32:07.858455 29048 external_mini_cluster.cc:1428] Setting key 13007fefc56dc72e6e87392d0e88230c
I20260806 04:32:07.861213 15470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap starting.
I20260806 04:32:07.863718 15470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:07.864583 15470 log.cc:824] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:07.866447 15470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: No bootstrap required, opened a new log
Aug 06 04:32:07 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:07.869558 15470 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:07.869777 15470 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:07.869836 15470 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Initialized, Role: FOLLOWER
I20260806 04:32:07.870282 15470 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:07.870447 15470 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:07.870517 15470 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:07.870613 15470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:07.871614 15470 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:07.871939 15470 leader_election.cc:302] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd301212a9a84edf8f8528ec773394d4; no voters:
I20260806 04:32:07.872210 15470 leader_election.cc:288] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:32:07.872339 15475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:07.872381 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:07.859735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.28.94.62:46715)
0806 04:32:07.860124 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:07.860140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:07.860177 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:32:07.860846 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:07.860863 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:07.861233 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:07.861515 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:07.862356 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:07.863329 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:07.864428 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:07.864951 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:07.865102 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:07.867376 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:07.867382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:07.867395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:07.867421 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:07.869502 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:07.869914 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:07.869921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:07.869926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:07.870064 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:07.870367 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:07.870369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:07.870370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:07.870830 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:07.870973 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:07.871276 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 04:32:07.871565 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260806 04:32:07.872551 15475 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 LEADER]: Becoming Leader. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Running, Role: LEADER
I20260806 04:32:07.872864 15475 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:07.873412 15470 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:07.874082 15477 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd301212a9a84edf8f8528ec773394d4. Latest consensus state: current_term: 1 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:07.874198 15477 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:07.874902 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:07.874984 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
W20260806 04:32:07.878218 15495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:32:07.878285 15495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:32:07.878355 15482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:07.879058 15482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:07.884460 15482 catalog_manager.cc:1347] Generated new cluster ID: 02d97a847d0a40d7bdb3360f66bae0f5
I20260806 04:32:07.884521 15482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:07.906291 15482 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:32:07.907485 15482 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:07.913825 15482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Generated new TSK 0
I20260806 04:32:07.914461 15482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:32:07.954070 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:08.068810 15505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:08.069092 15505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:08.069130 15505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:08.073004 15505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:08.073057 15505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:08.073114 15505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:32:08.078276 15505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:08.079250 15505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:08.080206 15505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:08.089526 15510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:08.089551 15517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:08.089535 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:08.089790 15505 server_base.cc:1034] running on GCE node
I20260806 04:32:08.090196 15505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:08.090816 15505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:08.091970 15505 hybrid_clock.cc:648] HybridClock initialized: now 1785990728091953 us; error 27 us; skew 500 ppm
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:08.094651 15505 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:32:08.095549 15505 webserver.cc:460] Webserver started at http://127.28.94.1:33769/ using document root <none> and password file <none>
I20260806 04:32:08.096105 15505 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:08.096170 15505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:08.096377 15505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:08.098384 15505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "a420562b4f05446199b5bcf1cdf50149"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "32825b2fb08f8a7ea6ebcf7a9d4d746e"
server_key_iv: "3feed1bdc47a7955bcaa90535bb0e0d3"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.098927 15505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "a420562b4f05446199b5bcf1cdf50149"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "32825b2fb08f8a7ea6ebcf7a9d4d746e"
server_key_iv: "3feed1bdc47a7955bcaa90535bb0e0d3"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.102535 15505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 04:32:08.104852 15524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.106029 15505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:32:08.106149 15505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "a420562b4f05446199b5bcf1cdf50149"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "32825b2fb08f8a7ea6ebcf7a9d4d746e"
server_key_iv: "3feed1bdc47a7955bcaa90535bb0e0d3"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.106240 15505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:08.133427 15505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:08.136070 15505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:08.136258 15505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:08.136835 15505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:08.137684 15505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:08.137732 15505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.137786 15505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:08.137814 15505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.146109 15505 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:39105
I20260806 04:32:08.146137 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:39105 every 8 connection(s)
I20260806 04:32:08.146955 15505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:32:08.150857 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 15505
I20260806 04:32:08.150985 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:32:08.151255 29048 external_mini_cluster.cc:1428] Setting key 18a871059aa5a0548cc1e550b7675e44
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:08.160040 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.148713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:49589 (local address 127.28.94.62:46715)
0806 04:32:08.148855 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.148859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.149686 (+ 827us) server_negotiation.cc:407] Connection header received
0806 04:32:08.151000 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.151004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.151062 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.151137 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:08.152376 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.153038 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.154144 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.154370 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.154457 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.156735 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:08.156738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:08.156740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:08.156765 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:08.158122 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.158601 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.158604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.158605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.158684 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.158989 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.158992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.158994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.159179 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:08.159264 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.159748 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.159875 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:32:08.160928 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.149086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:49589)
0806 04:32:08.149538 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.149580 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.150793 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.151326 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.151335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.151742 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:08.152236 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.152250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.153168 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.153171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.154030 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.154040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.154507 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.154512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.154668 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.155196 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:08.155212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:08.156573 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:08.158240 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.158246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.158260 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.158507 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.158765 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.158769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.158771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.158902 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.159278 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:08.159285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:08.159517 (+ 232us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.159690 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.160000 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260806 04:32:08.162066 15679 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:08.162292 15679 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:08.162765 15679 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:32:08.164261 15412 ts_manager.cc:194] Registered new tserver with Master: a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105)
I20260806 04:32:08.165793 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:49589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:32:08.179203 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:08.291364 15686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:08.291615 15686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:08.291662 15686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:08.295629 15686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:08.295682 15686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:08.295753 15686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:32:08.300844 15686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:08.301882 15686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:08.302927 15686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:08.312343 15699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:08.312354 15694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:08.312394 15696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:08.312634 15686 server_base.cc:1034] running on GCE node
I20260806 04:32:08.313071 15686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:08.313678 15686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:08.314847 15686 hybrid_clock.cc:648] HybridClock initialized: now 1785990728314816 us; error 37 us; skew 500 ppm
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:08.317674 15686 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:32:08.318480 15686 webserver.cc:460] Webserver started at http://127.28.94.2:35971/ using document root <none> and password file <none>
I20260806 04:32:08.318989 15686 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:08.319046 15686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:08.319226 15686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:08.321262 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "dc32d27d1cfcd497725ec1d6969e0058"
server_key_iv: "888e0af7e1dd08fad72d8aa314789f47"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.321767 15686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "dc32d27d1cfcd497725ec1d6969e0058"
server_key_iv: "888e0af7e1dd08fad72d8aa314789f47"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.325336 15686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:32:08.327651 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.328789 15686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:32:08.328917 15686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "dc32d27d1cfcd497725ec1d6969e0058"
server_key_iv: "888e0af7e1dd08fad72d8aa314789f47"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.329011 15686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:08.343755 15686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:08.346410 15686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:08.346578 15686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:08.347225 15686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:08.348050 15686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:08.348096 15686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.348142 15686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:08.348171 15686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.356627 15686 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42207
I20260806 04:32:08.356654 15871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42207 every 8 connection(s)
I20260806 04:32:08.357542 15686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:32:08.365408 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 15686
I20260806 04:32:08.365536 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:32:08.365796 29048 external_mini_cluster.cc:1428] Setting key f618f85736d6febd5874ebfcbcb42a72
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:08.370642 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.359282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:45911 (local address 127.28.94.62:46715)
0806 04:32:08.359420 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.359423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.360235 (+ 812us) server_negotiation.cc:407] Connection header received
0806 04:32:08.361609 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.361612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.361663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.361736 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:08.362840 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.363355 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.364381 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.364588 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.364654 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.367154 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:08.367158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:08.367160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:08.367187 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:08.368579 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.369088 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.369092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.369093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.369192 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.369547 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.369550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.369551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.369740 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:08.369831 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.370320 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.370450 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:32:08.371543 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.359624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:45911)
0806 04:32:08.360084 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.360128 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.361408 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.361881 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.361887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.362260 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:08.362728 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.362741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.363438 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.363441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.364293 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.364301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.364689 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.364691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.364823 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.365404 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:08.365420 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:08.366984 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:08.368687 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.368692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.368703 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.368999 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.369267 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.369269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.369270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.369438 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.369843 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:08.369850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:08.370083 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.370238 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.370574 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260806 04:32:08.372781 15873 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:08.373070 15873 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:08.373674 15873 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
I20260806 04:32:08.374810 15412 ts_manager.cc:194] Registered new tserver with Master: 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
I20260806 04:32:08.375473 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:45911
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:32:08.393285 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:08.506672 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:08.506942 15880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:08.507006 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:08.511294 15880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:08.511358 15880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:08.511430 15880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:32:08.516757 15880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:08.517856 15880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:08.518944 15880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:08.528633 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:08.528738 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:08.528824 15880 server_base.cc:1034] running on GCE node
W20260806 04:32:08.528693 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:08.529320 15880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:08.529939 15880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:08.531103 15880 hybrid_clock.cc:648] HybridClock initialized: now 1785990728531081 us; error 34 us; skew 500 ppm
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:08.533892 15880 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:32:08.534772 15880 webserver.cc:460] Webserver started at http://127.28.94.3:37797/ using document root <none> and password file <none>
I20260806 04:32:08.535329 15880 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:08.535395 15880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:08.535604 15880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:08.537688 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "ef8373ef02764282a7ae04ceae475ea0"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "94b83d2c9d52a59a08d9a97f6576711e"
server_key_iv: "c944f0042629bf3aa9fde635f48b00bd"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.538223 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "ef8373ef02764282a7ae04ceae475ea0"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "94b83d2c9d52a59a08d9a97f6576711e"
server_key_iv: "c944f0042629bf3aa9fde635f48b00bd"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.541831 15880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:32:08.544111 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.545359 15880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:32:08.545485 15880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "ef8373ef02764282a7ae04ceae475ea0"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "94b83d2c9d52a59a08d9a97f6576711e"
server_key_iv: "c944f0042629bf3aa9fde635f48b00bd"
server_key_version: "encryptionkey@0"
I20260806 04:32:08.545581 15880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:08.569942 15880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:08.573249 15880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:08.573443 15880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:08.574110 15880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:08.575006 15880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:08.575057 15880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.575114 15880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:08.575138 15880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:08.583421 15880 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:35855
I20260806 04:32:08.583451 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:35855 every 8 connection(s)
I20260806 04:32:08.584297 15880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:32:08.589673 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 15880
I20260806 04:32:08.589777 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:32:08.590018 29048 external_mini_cluster.cc:1428] Setting key be921706b7788fb022f383554f5c5b34
Aug 06 04:32:08 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:08.597282 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.586078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:44971 (local address 127.28.94.62:46715)
0806 04:32:08.586219 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.586222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.587026 (+ 804us) server_negotiation.cc:407] Connection header received
0806 04:32:08.588284 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.588287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.588336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.588405 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:08.589551 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.590152 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.591434 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.591656 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.591731 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.594075 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:08.594078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:08.594080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:08.594105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:08.595356 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.595831 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.595834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.595835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.595911 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.596220 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.596224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.596225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.596400 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:08.596486 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.597003 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.597121 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:32:08.598048 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.586427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:44971)
0806 04:32:08.586872 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.586913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.588097 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.588548 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.588555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.588924 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:08.589399 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.589416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.590273 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.590276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.591315 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.591326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.591793 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.591797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.591981 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.592496 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:08.592512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:08.593916 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:08.595466 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.595471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.595486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.595735 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.595995 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:08.595998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:08.596000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:08.596131 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:08.596510 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:08.596517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:08.596746 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.596930 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.597240 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260806 04:32:08.599057 16068 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:08.599282 16068 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:08.599759 16068 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
I20260806 04:32:08.600739 15412 ts_manager.cc:194] Registered new tserver with Master: ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855)
I20260806 04:32:08.601327 15412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:44971
I20260806 04:32:08.603737 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:32:08.611927 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.605698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.28.94.62:46715)
0806 04:32:08.605823 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.605827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.606023 (+ 196us) server_negotiation.cc:407] Connection header received
0806 04:32:08.606119 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.606121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.606159 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.606219 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:08.606838 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.607313 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.608061 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.608274 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.608337 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.609545 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:08.609548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:08.609550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:08.609573 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:08.610677 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.611023 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.611025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.611026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.611090 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:08.611262 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:08.611264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:08.611265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:08.611450 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:08.611561 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.611701 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.611786 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:32:08.614475 15412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:32:08.617758 15412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:32:08.631585 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.626077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.0.0.1:36756)
0806 04:32:08.626591 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.626639 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.626769 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.627321 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.627327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.627352 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:08.627492 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.627499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.628850 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.628853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.630257 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.630268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.631091 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.631094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.631230 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.631275 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.631372 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.631467 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260806 04:32:08.632072 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.626225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.28.94.1:39105)
0806 04:32:08.626885 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.626901 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.626927 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:32:08.627005 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.627011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.627180 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.627337 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:08.627603 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.628721 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.630377 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.630974 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.631088 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.631237 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.631869 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.631959 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":93,"threads_started":1}
I20260806 04:32:08.632490 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.626774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:35855 (local address 127.0.0.1:41538)
0806 04:32:08.627861 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.627872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.627950 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.628374 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.628377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.628394 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:08.628496 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.628502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.629621 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.629624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.630734 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.630745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.632112 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.632116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.632207 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.632230 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.632301 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.632365 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260806 04:32:08.632645 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.627520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.28.94.3:35855)
0806 04:32:08.627840 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.627851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.627931 (+ 80us) server_negotiation.cc:407] Connection header received
0806 04:32:08.628036 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.628041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.628232 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.628374 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:08.628601 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.629469 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.630866 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.631720 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.632097 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.632239 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.632331 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.632424 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260806 04:32:08.632876 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.626245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42207 (local address 127.0.0.1:36050)
0806 04:32:08.626999 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.627012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.627096 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.628702 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.628704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.628719 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:08.628787 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.628791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.630201 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.630205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.631248 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.631258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.632463 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.632467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.632592 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.632615 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.632687 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.632755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":24,"thread_start_us":118,"threads_started":1}
I20260806 04:32:08.633282 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.627487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.28.94.2:42207)
0806 04:32:08.628157 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.628166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.628183 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:32:08.628244 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.628247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.628402 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.628543 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:08.628933 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.630067 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.631355 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.632142 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.632232 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.632355 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.633066 (+ 711us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.633159 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":144,"threads_started":1}
I20260806 04:32:08.635516 15773 tablet_service.cc:1467] Processing CreateTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 (DEFAULT_TABLE table=test-table [id=82d456eb0ea241e7a05f47bc2f67f462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:08.635648 15968 tablet_service.cc:1467] Processing CreateTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 (DEFAULT_TABLE table=test-table [id=82d456eb0ea241e7a05f47bc2f67f462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:08.635532 15590 tablet_service.cc:1467] Processing CreateTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 (DEFAULT_TABLE table=test-table [id=82d456eb0ea241e7a05f47bc2f67f462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:08.636497 15773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fb56ee6693472cbfa503cf3df3a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:08.636556 15590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fb56ee6693472cbfa503cf3df3a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:08.636565 15968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fb56ee6693472cbfa503cf3df3a338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:08.642761 16092 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Bootstrap starting.
I20260806 04:32:08.644073 16093 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Bootstrap starting.
I20260806 04:32:08.644284 16094 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Bootstrap starting.
I20260806 04:32:08.645457 16092 tablet_bootstrap.cc:654] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:08.646185 16092 log.cc:824] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:08.646296 16094 tablet_bootstrap.cc:654] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:08.646296 16093 tablet_bootstrap.cc:654] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:08.647586 16094 log.cc:824] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:08.647586 16093 log.cc:824] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:08.648234 16092 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: No bootstrap required, opened a new log
I20260806 04:32:08.648444 16092 ts_tablet_manager.cc:1397] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 04:32:08.649515 16094 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: No bootstrap required, opened a new log
I20260806 04:32:08.649518 16093 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: No bootstrap required, opened a new log
I20260806 04:32:08.649705 16093 ts_tablet_manager.cc:1397] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 04:32:08.649705 16094 ts_tablet_manager.cc:1397] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:32:08.651575 16092 raft_consensus.cc:348] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.651788 16092 raft_consensus.cc:374] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:08.651859 16092 raft_consensus.cc:729] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a0bf03f31e4b62979e3591aa6b14cb, State: Initialized, Role: FOLLOWER
I20260806 04:32:08.652285 16092 consensus_queue.cc:260] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.652984 15873 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
I20260806 04:32:08.653260 16092 ts_tablet_manager.cc:1428] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260806 04:32:08.653424 16093 raft_consensus.cc:348] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.653681 16093 raft_consensus.cc:374] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:08.653774 16093 raft_consensus.cc:729] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8373ef02764282a7ae04ceae475ea0, State: Initialized, Role: FOLLOWER
I20260806 04:32:08.653929 16094 raft_consensus.cc:348] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.654232 16094 raft_consensus.cc:374] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:08.654318 16094 raft_consensus.cc:729] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420562b4f05446199b5bcf1cdf50149, State: Initialized, Role: FOLLOWER
I20260806 04:32:08.654326 16093 consensus_queue.cc:260] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.654776 16094 consensus_queue.cc:260] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.655086 16068 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
I20260806 04:32:08.655367 16093 ts_tablet_manager.cc:1428] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:32:08.655853 16094 ts_tablet_manager.cc:1428] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:32:08.656173 15679 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
I20260806 04:32:08.792232 16101 raft_consensus.cc:482] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:08.792455 16101 raft_consensus.cc:504] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.793697 16101 leader_election.cc:288] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
I20260806 04:32:08.797178 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.794033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:38563 (local address 127.28.94.1:39105)
0806 04:32:08.794160 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.794165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.794179 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:32:08.794359 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.794362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.794415 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.794497 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:08.794764 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.795253 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796252 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.796633 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796734 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.796801 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.796981 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.797049 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:32:08.797178 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.793952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.28.94.3:38563)
0806 04:32:08.794069 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.794083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.794193 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.794530 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.794533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.794552 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:08.794657 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.794662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.795356 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.795360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.796120 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.796128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.796751 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.796854 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.796873 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.796964 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.797040 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 04:32:08.797351 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.793959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42207 (local address 127.28.94.3:46165)
0806 04:32:08.794377 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.794389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:08.794488 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:08.794720 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:08.794723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:08.794738 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:08.794849 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.794854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.795538 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.795541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.796308 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:08.796317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.796952 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:08.797060 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.797079 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:32:08.797145 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:32:08.797210 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":44,"thread_start_us":164,"threads_started":1}
I20260806 04:32:08.797341 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.794219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:46165 (local address 127.28.94.2:42207)
0806 04:32:08.794367 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.794371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.794461 (+ 90us) server_negotiation.cc:407] Connection header received
0806 04:32:08.794586 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.794590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.794634 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.794697 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:08.794948 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.795435 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796428 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.796824 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.796938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.797000 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.797170 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.797238 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:32:08.797868 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fb56ee6693472cbfa503cf3df3a338" candidate_uuid: "ef8373ef02764282a7ae04ceae475ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a420562b4f05446199b5bcf1cdf50149" is_pre_election: true
I20260806 04:32:08.798017 15799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fb56ee6693472cbfa503cf3df3a338" candidate_uuid: "ef8373ef02764282a7ae04ceae475ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" is_pre_election: true
I20260806 04:32:08.798166 15619 raft_consensus.cc:2454] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8373ef02764282a7ae04ceae475ea0 in term 0.
I20260806 04:32:08.798303 15799 raft_consensus.cc:2454] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef8373ef02764282a7ae04ceae475ea0 in term 0.
I20260806 04:32:08.798619 15908 leader_election.cc:302] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149, ef8373ef02764282a7ae04ceae475ea0; no voters:
I20260806 04:32:08.798884 16101 raft_consensus.cc:2781] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:08.798961 16101 raft_consensus.cc:482] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:08.798992 16101 raft_consensus.cc:3037] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:08.800006 16101 raft_consensus.cc:504] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.800335 16101 leader_election.cc:288] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [CANDIDATE]: Term 1 election: Requested vote from peers a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
I20260806 04:32:08.800654 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fb56ee6693472cbfa503cf3df3a338" candidate_uuid: "ef8373ef02764282a7ae04ceae475ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a420562b4f05446199b5bcf1cdf50149"
I20260806 04:32:08.800691 15799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fb56ee6693472cbfa503cf3df3a338" candidate_uuid: "ef8373ef02764282a7ae04ceae475ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
I20260806 04:32:08.800782 15619 raft_consensus.cc:3037] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:08.800787 15799 raft_consensus.cc:3037] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:08.801786 15799 raft_consensus.cc:2454] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8373ef02764282a7ae04ceae475ea0 in term 1.
I20260806 04:32:08.801805 15619 raft_consensus.cc:2454] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef8373ef02764282a7ae04ceae475ea0 in term 1.
I20260806 04:32:08.802047 15904 leader_election.cc:302] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0; no voters:
I20260806 04:32:08.802219 16101 raft_consensus.cc:2781] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:08.802415 16101 raft_consensus.cc:686] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 1 LEADER]: Becoming Leader. State: Replica: ef8373ef02764282a7ae04ceae475ea0, State: Running, Role: LEADER
I20260806 04:32:08.802740 16101 consensus_queue.cc:237] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:08.805927 15409 catalog_manager.cc:5495] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "ef8373ef02764282a7ae04ceae475ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } health_report { overall_health: HEALTHY } } }
I20260806 04:32:08.818288 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.815256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.28.94.3:35855)
0806 04:32:08.815372 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:08.815375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:08.815388 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:32:08.815522 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:08.815524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:08.815590 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:08.815668 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:08.815835 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.816309 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.817100 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:08.817305 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:08.817377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:08.817582 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:08.817973 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:08.818057 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:08.818081 (+ 24us) server_negotiation.cc:299] Negotiation successful
0806 04:32:08.818138 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:32:08.824177 15619 raft_consensus.cc:1261] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Refusing update from remote peer ef8373ef02764282a7ae04ceae475ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:08.824177 15799 raft_consensus.cc:1261] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Refusing update from remote peer ef8373ef02764282a7ae04ceae475ea0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:08.824971 16105 consensus_queue.cc:1035] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:08.825148 16101 consensus_queue.cc:1035] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:08.833647 16114 mvcc.cc:205] Tried to move back new op lower bound from 7315418025256271872 to 7315418025178554368. Current Snapshot: MvccSnapshot[applied={T|T < 7315418025256271872}]
I20260806 04:32:08.834072 16113 mvcc.cc:205] Tried to move back new op lower bound from 7315418025256271872 to 7315418025178554368. Current Snapshot: MvccSnapshot[applied={T|T < 7315418025256271872}]
I20260806 04:32:08.840133 15412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:08.840328 15412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:08.842976 15412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Sending DeleteTablet for 3 replicas of tablet e0fb56ee6693472cbfa503cf3df3a338
I20260806 04:32:08.843822 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 15505
I20260806 04:32:08.843780 15590 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:08 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:36756
I20260806 04:32:08.843837 15773 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:08 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:36050
I20260806 04:32:08.846288 15968 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:08 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260806 04:32:08.848140 16124 tablet_replica.cc:330] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: stopping tablet replica
I20260806 04:32:08.848536 16124 raft_consensus.cc:2229] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:08.848946 16124 raft_consensus.cc:2258] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:08.852802 16124 ts_tablet_manager.cc:1905] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:08.855163 16124 ts_tablet_manager.cc:1918] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:08.855268 16124 log.cc:1196] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/e0fb56ee6693472cbfa503cf3df3a338
I20260806 04:32:08.855621 16124 ts_tablet_manager.cc:1939] T e0fb56ee6693472cbfa503cf3df3a338 P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting consensus metadata
I20260806 04:32:08.855633 16126 tablet_replica.cc:330] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: stopping tablet replica
I20260806 04:32:08.856003 16126 raft_consensus.cc:2229] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:32:08.856539 16126 raft_consensus.cc:2258] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:08.857200 15389 catalog_manager.cc:4841] TS 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): tablet e0fb56ee6693472cbfa503cf3df3a338 (table test-table [id=82d456eb0ea241e7a05f47bc2f67f462]) successfully deleted
I20260806 04:32:08.859579 16126 ts_tablet_manager.cc:1905] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:32:08.861052 15394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:39105 (error 108)
I20260806 04:32:08.862064 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 15686
I20260806 04:32:08.879619 16126 ts_tablet_manager.cc:1918] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:08.879750 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:08.879289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.0.0.1:37760)
0806 04:32:08.879532 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:08.879650 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:32:08.879815 16126 log.cc:1196] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/e0fb56ee6693472cbfa503cf3df3a338
W20260806 04:32:08.879969 15394 catalog_manager.cc:4584] TS a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0fb56ee6693472cbfa503cf3df3a338: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39105: connect: Connection refused (error 111)
I20260806 04:32:08.880277 16126 ts_tablet_manager.cc:1939] T e0fb56ee6693472cbfa503cf3df3a338 P ef8373ef02764282a7ae04ceae475ea0: Deleting consensus metadata
I20260806 04:32:08.881749 15389 catalog_manager.cc:4841] TS ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): tablet e0fb56ee6693472cbfa503cf3df3a338 (table test-table [id=82d456eb0ea241e7a05f47bc2f67f462]) successfully deleted
I20260806 04:32:08.885344 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 15880
I20260806 04:32:08.899580 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 15366
I20260806 04:32:08.914036 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--webserver_interface=127.28.94.62
--webserver_port=39717
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:09.030056 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:09.030337 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:09.030380 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:09.034413 16130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:09.034477 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:09.034502 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:09.034516 16130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:09.034531 16130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:09.039907 16130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=39717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:09.041075 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:09.042155 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:32:09.057742 16130 server_base.cc:1034] running on GCE node
W20260806 04:32:09.057641 16140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.057647 16143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.057641 16139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:09.058660 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:09.059911 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:09.061100 16130 hybrid_clock.cc:648] HybridClock initialized: now 1785990729061071 us; error 36 us; skew 500 ppm
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:09.064497 16130 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:09.065519 16130 webserver.cc:460] Webserver started at http://127.28.94.62:39717/ using document root <none> and password file <none>
I20260806 04:32:09.066114 16130 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:09.066299 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:09.070659 16130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:09.073025 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.074340 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:09.074476 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "fd301212a9a84edf8f8528ec773394d4"
format_stamp: "Formatted at 2026-08-06 04:32:07 on dist-test-slave-ncp9"
server_key: "392a55c5ef47ed0444ad130724a20926"
server_key_iv: "40fd3066abd95bd4d167405fd939cabc"
server_key_version: "encryptionkey@0"
I20260806 04:32:09.075114 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:09.088498 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:09.091658 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:09.091828 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:09.099246 16130 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46715
I20260806 04:32:09.099560 16231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46715 every 8 connection(s)
I20260806 04:32:09.100412 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:09.101239 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16130
I20260806 04:32:09.101657 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:39105
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=33769
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:09.105401 16233 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:32:09.107743 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap starting.
I20260806 04:32:09.131999 16233 log.cc:824] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:09.143532 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:09.143992 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap complete.
I20260806 04:32:09.149785 16233 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:09.150321 16233 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Initialized, Role: FOLLOWER
I20260806 04:32:09.150918 16233 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:09.151067 16233 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:09.151142 16233 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:09.151237 16233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:32:09.153892 16233 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:09.154284 16233 leader_election.cc:302] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd301212a9a84edf8f8528ec773394d4; no voters:
I20260806 04:32:09.154676 16233 leader_election.cc:288] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:32:09.154806 16238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:32:09.155082 16238 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 LEADER]: Becoming Leader. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Running, Role: LEADER
I20260806 04:32:09.155411 16238 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:09.156044 16233 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:09.156912 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd301212a9a84edf8f8528ec773394d4. Latest consensus state: current_term: 2 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:09.157060 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:09.156939 16239 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:09.157305 16239 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:09.161863 16263 catalog_manager.cc:1259] Loaded cluster ID: 02d97a847d0a40d7bdb3360f66bae0f5
I20260806 04:32:09.161924 16263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: loading cluster ID for follower catalog manager: success
W20260806 04:32:09.162923 16263 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260806 04:32:09.162997 16263 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260806 04:32:09.163101 16256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:09.165701 16256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:09.166208 16256 catalog_manager.cc:1259] Loaded cluster ID: 02d97a847d0a40d7bdb3360f66bae0f5
I20260806 04:32:09.166268 16256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:09.167300 16256 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 04:32:09.171782 16256 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 04:32:09.177619 16256 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:09.178473 16256 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Loaded TSK: 0
I20260806 04:32:09.179811 16256 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:32:09.243518 16235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:09.243780 16235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:09.243824 16235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:09.247900 16235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:09.247974 16235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:09.248035 16235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:32:09.253391 16235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:39105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=33769
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:09.254603 16235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:09.255641 16235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:09.265836 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.265836 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.266116 16276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:09.266160 16235 server_base.cc:1034] running on GCE node
I20260806 04:32:09.266820 16235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:09.267707 16235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:09.268972 16235 hybrid_clock.cc:648] HybridClock initialized: now 1785990729268942 us; error 42 us; skew 500 ppm
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:09.272631 16235 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:32:09.274010 16235 webserver.cc:460] Webserver started at http://127.28.94.1:33769/ using document root <none> and password file <none>
I20260806 04:32:09.274847 16235 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:09.274968 16235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:09.280421 16235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:09.283077 16284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.284621 16235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:09.284767 16235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "a420562b4f05446199b5bcf1cdf50149"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "32825b2fb08f8a7ea6ebcf7a9d4d746e"
server_key_iv: "3feed1bdc47a7955bcaa90535bb0e0d3"
server_key_version: "encryptionkey@0"
I20260806 04:32:09.285514 16235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:09.304859 16235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:09.308588 16235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:09.308792 16235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:09.309569 16235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:09.310915 16297 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:32:09.315433 16235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:32:09.315503 16235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 04:32:09.315560 16235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:32:09.317370 16235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:32:09.317420 16235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:09.317737 16297 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Bootstrap starting.
I20260806 04:32:09.326819 16235 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:39105
I20260806 04:32:09.326889 16447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:39105 every 8 connection(s)
I20260806 04:32:09.327800 16235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:32:09.330786 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16235
I20260806 04:32:09.331295 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:42207
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=35971
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:09.338176 16297 log.cc:824] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:09.345820 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.330089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:33099)
0806 04:32:09.330638 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.330695 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.331930 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.333098 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.333106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.333544 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:09.334026 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.334041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.335346 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.335349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.336235 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.336245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.336968 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.336972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.337181 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.337849 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:09.337869 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:09.339546 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:09.342109 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.342116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.342131 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.342506 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.342929 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.342932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.342934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.343126 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.343912 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:09.343919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:09.344227 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.344423 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.344772 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
I20260806 04:32:09.345877 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.330165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:33099 (local address 127.28.94.62:46715)
0806 04:32:09.331095 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.331120 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.331162 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:32:09.332367 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.332377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.332682 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.333047 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:09.334209 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.335190 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.336360 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.336821 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.336996 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.339780 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:09.339786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:09.339800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:09.339832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:09.341966 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.342645 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.342651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.342658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.342821 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.343254 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.343258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.343259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.343795 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:09.343944 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.344586 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.344967 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":133,"threads_started":1}
I20260806 04:32:09.348328 16297 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:09.348508 16449 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:09.348752 16449 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:09.348763 16297 tablet_bootstrap.cc:492] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Bootstrap complete.
I20260806 04:32:09.349398 16449 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
I20260806 04:32:09.350208 16297 ts_tablet_manager.cc:1397] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.020s
I20260806 04:32:09.352342 16179 ts_manager.cc:194] Registered new tserver with Master: a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105)
I20260806 04:32:09.352702 16297 raft_consensus.cc:348] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:09.353188 16297 raft_consensus.cc:729] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420562b4f05446199b5bcf1cdf50149, State: Initialized, Role: FOLLOWER
I20260806 04:32:09.353631 16297 consensus_queue.cc:260] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } }
I20260806 04:32:09.354565 16297 ts_tablet_manager.cc:1428] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260806 04:32:09.356498 16179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:33099
I20260806 04:32:09.359465 16449 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:32:09.366799 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.355844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.0.0.1:37776)
0806 04:32:09.356269 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.356315 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.356449 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.357451 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.357458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.357868 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:09.358026 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.358036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.359741 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.359744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.360565 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.360577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.361260 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.361264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.361377 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.361922 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:09.361934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:09.363323 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:09.365067 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.365076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.365082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.365302 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.365595 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.365598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.365600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.365691 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.366134 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:09.366141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:09.366264 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.366383 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.366603 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":135,"threads_started":1}
I20260806 04:32:09.366799 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.356603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.28.94.1:39105)
0806 04:32:09.356942 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.356955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.356981 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:32:09.357071 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.357075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.357273 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.358199 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:09.358614 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.359603 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.360696 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.361115 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.361260 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.363458 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:09.363468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:09.363483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:09.363507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:09.364957 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.365410 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.365416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.365418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.365502 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.365772 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.365775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.365777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.366013 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:09.366127 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.366479 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.366619 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260806 04:32:09.367847 16355 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fb56ee6693472cbfa503cf3df3a338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:08 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:37776
I20260806 04:32:09.368710 16464 tablet_replica.cc:330] stopping tablet replica
I20260806 04:32:09.369055 16464 raft_consensus.cc:2229] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:09.369304 16464 raft_consensus.cc:2258] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:09.370311 16464 ts_tablet_manager.cc:1905] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:09.373961 16464 ts_tablet_manager.cc:1918] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:09.374049 16464 log.cc:1196] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/e0fb56ee6693472cbfa503cf3df3a338
I20260806 04:32:09.374292 16464 ts_tablet_manager.cc:1939] T e0fb56ee6693472cbfa503cf3df3a338 P a420562b4f05446199b5bcf1cdf50149: Deleting consensus metadata
I20260806 04:32:09.376015 16162 catalog_manager.cc:4841] TS a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105): tablet e0fb56ee6693472cbfa503cf3df3a338 (table test-table [id=82d456eb0ea241e7a05f47bc2f67f462]) successfully deleted
W20260806 04:32:09.463296 16454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:09.463550 16454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:09.463594 16454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:09.467693 16454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:09.467759 16454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:09.467819 16454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:32:09.473312 16454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:42207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=35971
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:09.474435 16454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:09.475481 16454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:32:09.485347 16454 server_base.cc:1034] running on GCE node
W20260806 04:32:09.485257 16475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.485276 16471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.485276 16472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:09.485880 16454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:09.486497 16454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:09.487656 16454 hybrid_clock.cc:648] HybridClock initialized: now 1785990729487630 us; error 37 us; skew 500 ppm
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:09.490274 16454 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:32:09.491231 16454 webserver.cc:460] Webserver started at http://127.28.94.2:35971/ using document root <none> and password file <none>
I20260806 04:32:09.491855 16454 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:09.491945 16454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:09.495886 16454 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:09.498035 16485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.499225 16454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:32:09.499341 16454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "dc32d27d1cfcd497725ec1d6969e0058"
server_key_iv: "888e0af7e1dd08fad72d8aa314789f47"
server_key_version: "encryptionkey@0"
I20260806 04:32:09.499889 16454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:09.518482 16454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:09.522069 16454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:09.522287 16454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:09.522953 16454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:09.523856 16454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:09.523908 16454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.523968 16454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:09.523995 16454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.532758 16454 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42207
I20260806 04:32:09.532805 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42207 every 8 connection(s)
I20260806 04:32:09.533882 16454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:32:09.542934 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16454
I20260806 04:32:09.543366 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:35855
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=37797
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:09.549938 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.535807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:49293 (local address 127.28.94.62:46715)
0806 04:32:09.535937 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.535941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.536817 (+ 876us) server_negotiation.cc:407] Connection header received
0806 04:32:09.538208 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.538211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.538274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.538348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:09.539578 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.540258 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.541517 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.541766 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.541859 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.544603 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:09.544606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:09.544608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:09.544639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:09.547574 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.548115 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.548127 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.548128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.548224 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.548631 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.548635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.548637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.548936 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:09.549065 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.549585 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.549749 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:32:09.550853 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.536171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:49293)
0806 04:32:09.536659 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.536702 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.537980 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.538540 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.538547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.538920 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:09.539414 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.539428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.540421 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.540424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.541382 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.541393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.541921 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.541926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.542103 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.542734 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:09.542752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:09.544391 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:09.547712 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.547720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.547740 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.548011 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.548315 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.548319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.548321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.548510 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.549049 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:09.549055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:09.549334 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.549506 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.549803 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260806 04:32:09.552086 16647 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:09.552343 16647 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:09.552865 16647 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
I20260806 04:32:09.554054 16179 ts_manager.cc:194] Registered new tserver with Master: 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
I20260806 04:32:09.554883 16179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:49293
W20260806 04:32:09.663477 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:09.663755 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:09.663805 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:09.667925 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:09.667987 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:09.668047 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:32:09.673641 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=37797
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:09.674818 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:09.675860 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:09.685702 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:09.685873 16652 server_base.cc:1034] running on GCE node
W20260806 04:32:09.685701 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:09.686038 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:09.686468 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:09.687116 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:09.688271 16652 hybrid_clock.cc:648] HybridClock initialized: now 1785990729688243 us; error 43 us; skew 500 ppm
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:09.691170 16652 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:32:09.692190 16652 webserver.cc:460] Webserver started at http://127.28.94.3:37797/ using document root <none> and password file <none>
I20260806 04:32:09.692770 16652 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:09.692855 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:09.696947 16652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:09.699060 16673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.700340 16652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:09.700469 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "ef8373ef02764282a7ae04ceae475ea0"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "94b83d2c9d52a59a08d9a97f6576711e"
server_key_iv: "c944f0042629bf3aa9fde635f48b00bd"
server_key_version: "encryptionkey@0"
I20260806 04:32:09.701094 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:09.717743 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:09.721782 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:09.721992 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:09.722661 16652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:09.723565 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:09.723616 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.723672 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:09.723701 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:09.732402 16652 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:35855
I20260806 04:32:09.732430 16838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:35855 every 8 connection(s)
I20260806 04:32:09.733520 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:32:09.741819 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16652
Aug 06 04:32:09 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:09.747603 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.735361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:60195 (local address 127.28.94.62:46715)
0806 04:32:09.735546 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.735552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.736424 (+ 872us) server_negotiation.cc:407] Connection header received
0806 04:32:09.737757 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.737760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.737813 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.737886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:09.739081 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.739629 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.740733 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.741027 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.741122 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.743818 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:09.743822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:09.743825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:09.743853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:09.745455 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.746024 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.746027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.746028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.746111 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.746491 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.746495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.746496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.746707 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:09.746811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.747328 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.747453 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:32:09.748685 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.735779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:60195)
0806 04:32:09.736275 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.736320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.737558 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.738041 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.738048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.738419 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:09.738921 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.738935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.739734 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.739737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.740604 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.740614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.741164 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.741167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.741306 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.741892 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:09.741908 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:09.743651 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:09.745588 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.745594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.745608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.745926 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.746190 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:09.746193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:09.746195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:09.746387 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:09.746812 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:09.746818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:09.747080 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.747252 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.747565 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260806 04:32:09.750028 16839 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46715
I20260806 04:32:09.750294 16839 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:09.750775 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.742538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.28.94.62:46715)
0806 04:32:09.742793 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.742797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.742988 (+ 191us) server_negotiation.cc:407] Connection header received
0806 04:32:09.743131 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.743133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.743182 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.743261 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:09.743984 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.744642 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.745847 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.746157 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.746261 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.747505 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:09.747509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:09.747512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:09.747538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:09.749183 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.749598 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.749601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.749603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.749700 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:09.749930 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:09.749933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:09.749935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:09.750145 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:09.750306 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.750538 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.750650 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260806 04:32:09.750938 16839 heartbeater.cc:511] Master 127.28.94.62:46715 requested a full tablet report, sending...
I20260806 04:32:09.752089 16179 ts_manager.cc:194] Registered new tserver with Master: ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855)
I20260806 04:32:09.752684 16179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:60195
I20260806 04:32:09.764710 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:32:09.769193 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.765406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.28.94.62:46715)
0806 04:32:09.765587 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.765591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.765740 (+ 149us) server_negotiation.cc:407] Connection header received
0806 04:32:09.765901 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.765903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.766031 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.766118 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:09.766338 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.766872 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.767704 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.767947 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.768041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.768320 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:09.768787 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:09.768878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.768983 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.769046 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 04:32:09.769920 16179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:32:09.773342 16179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:32:09.781569 16355 tablet_service.cc:1467] Processing CreateTablet for tablet bfc9313fdf604babb8a5197e3ce6443b (DEFAULT_TABLE table=test-table [id=64bdccba937a4a2cade1af6b68ca1b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:09.782269 16355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc9313fdf604babb8a5197e3ce6443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:09.785121 16297 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Bootstrap starting.
I20260806 04:32:09.786100 16297 tablet_bootstrap.cc:654] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:09.786715 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.781175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.28.94.3:35855)
0806 04:32:09.781523 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.781534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.781555 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:32:09.781623 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.781628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.781768 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.781961 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:09.782244 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.783431 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.785190 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.786088 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.786190 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.786361 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.786469 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.786569 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":144,"threads_started":1}
I20260806 04:32:09.786703 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.780456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:35855 (local address 127.0.0.1:56876)
0806 04:32:09.780589 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.780603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.780687 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.781963 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.781966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.781988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:09.782106 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.782112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.783567 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.783571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.785042 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.785050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.786209 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.786213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.786314 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.786363 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.786436 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.786560 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":98}
I20260806 04:32:09.787050 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.780863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.28.94.2:42207)
0806 04:32:09.781336 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.781354 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.781381 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:32:09.781471 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.781478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.781746 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.781961 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:09.782476 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.783555 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.785175 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.786127 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.786578 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.786729 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.786782 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.786876 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":116,"threads_started":1}
I20260806 04:32:09.787117 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.780690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42207 (local address 127.0.0.1:39422)
0806 04:32:09.780985 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.781000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.781101 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.782242 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.782245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.782261 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:09.782344 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.782350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.783685 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.783688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.785037 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.785050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.786278 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.786282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.786392 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.786424 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.786474 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.786986 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":497,"thread_start_us":136,"threads_started":1}
I20260806 04:32:09.788189 16297 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: No bootstrap required, opened a new log
I20260806 04:32:09.788290 16297 ts_tablet_manager.cc:1397] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 04:32:09.788786 16297 raft_consensus.cc:348] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.788936 16297 raft_consensus.cc:374] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:09.788975 16297 raft_consensus.cc:729] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420562b4f05446199b5bcf1cdf50149, State: Initialized, Role: FOLLOWER
I20260806 04:32:09.789108 16297 consensus_queue.cc:260] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.789390 16297 ts_tablet_manager.cc:1428] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:32:09.790506 16744 tablet_service.cc:1467] Processing CreateTablet for tablet bfc9313fdf604babb8a5197e3ce6443b (DEFAULT_TABLE table=test-table [id=64bdccba937a4a2cade1af6b68ca1b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:09.790925 16551 tablet_service.cc:1467] Processing CreateTablet for tablet bfc9313fdf604babb8a5197e3ce6443b (DEFAULT_TABLE table=test-table [id=64bdccba937a4a2cade1af6b68ca1b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:09.791571 16744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc9313fdf604babb8a5197e3ce6443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:09.791898 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc9313fdf604babb8a5197e3ce6443b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:09.798560 16858 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Bootstrap starting.
I20260806 04:32:09.798599 16859 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Bootstrap starting.
I20260806 04:32:09.801100 16859 tablet_bootstrap.cc:654] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:09.801311 16858 tablet_bootstrap.cc:654] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:09.801970 16859 log.cc:824] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:09.802315 16858 log.cc:824] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:09.804155 16858 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: No bootstrap required, opened a new log
I20260806 04:32:09.804179 16859 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: No bootstrap required, opened a new log
I20260806 04:32:09.804373 16858 ts_tablet_manager.cc:1397] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:32:09.804373 16859 ts_tablet_manager.cc:1397] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:32:09.807550 16859 raft_consensus.cc:348] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.807554 16858 raft_consensus.cc:348] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.807832 16859 raft_consensus.cc:374] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:09.807832 16858 raft_consensus.cc:374] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:09.807895 16858 raft_consensus.cc:729] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a0bf03f31e4b62979e3591aa6b14cb, State: Initialized, Role: FOLLOWER
I20260806 04:32:09.807912 16859 raft_consensus.cc:729] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8373ef02764282a7ae04ceae475ea0, State: Initialized, Role: FOLLOWER
I20260806 04:32:09.808390 16859 consensus_queue.cc:260] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.808390 16858 consensus_queue.cc:260] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.809118 16839 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
I20260806 04:32:09.809126 16647 heartbeater.cc:503] Master 127.28.94.62:46715 was elected leader, sending a full tablet report...
I20260806 04:32:09.809511 16859 ts_tablet_manager.cc:1428] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:32:09.809518 16858 ts_tablet_manager.cc:1428] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260806 04:32:09.829042 16450 tablet.cc:2369] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:09.930527 16863 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:09.930796 16863 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.932122 16863 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105)
I20260806 04:32:09.935902 16856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.932602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:45801 (local address 127.28.94.3:35855)
0806 04:32:09.932722 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.932728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.932747 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:32:09.932808 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.932811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.932866 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.932975 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:09.933309 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.933824 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.934899 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.935317 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.935426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.935511 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.935693 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.935773 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 04:32:09.935902 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.932446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:35855 (local address 127.28.94.2:45801)
0806 04:32:09.932586 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.932603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.932712 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.933009 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.933013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.933038 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:09.933170 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.933180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.933958 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.933962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.934754 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.934764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.935449 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.935454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.935567 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.935589 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.935667 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.935744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":4}
I20260806 04:32:09.936548 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.932465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.28.94.2:42949)
0806 04:32:09.933217 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:09.933228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:09.933320 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:09.933593 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:09.933596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:09.933613 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:09.933722 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.933727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.934429 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.934433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.935201 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:09.935210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.936156 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.936159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:09.936254 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.936279 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:32:09.936340 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 04:32:09.936404 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":329,"thread_start_us":137,"threads_started":1}
I20260806 04:32:09.936762 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc9313fdf604babb8a5197e3ce6443b" candidate_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8373ef02764282a7ae04ceae475ea0" is_pre_election: true
I20260806 04:32:09.936951 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.932572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:42949 (local address 127.28.94.1:39105)
0806 04:32:09.932915 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.932920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.933313 (+ 393us) server_negotiation.cc:407] Connection header received
0806 04:32:09.933410 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.933413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.933470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.933567 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:09.933830 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.934323 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.935325 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.935785 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.935884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.936071 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.936712 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.936799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260806 04:32:09.937156 16774 raft_consensus.cc:2454] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57a0bf03f31e4b62979e3591aa6b14cb in term 0.
I20260806 04:32:09.937561 16387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc9313fdf604babb8a5197e3ce6443b" candidate_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a420562b4f05446199b5bcf1cdf50149" is_pre_election: true
I20260806 04:32:09.937691 16486 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0; no voters:
I20260806 04:32:09.937853 16387 raft_consensus.cc:2454] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57a0bf03f31e4b62979e3591aa6b14cb in term 0.
I20260806 04:32:09.938025 16863 raft_consensus.cc:2781] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:09.938124 16863 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:09.938211 16863 raft_consensus.cc:3037] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:09.939359 16863 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.939750 16863 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [CANDIDATE]: Term 1 election: Requested vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105)
I20260806 04:32:09.940193 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc9313fdf604babb8a5197e3ce6443b" candidate_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef8373ef02764282a7ae04ceae475ea0"
I20260806 04:32:09.940192 16387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc9313fdf604babb8a5197e3ce6443b" candidate_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a420562b4f05446199b5bcf1cdf50149"
I20260806 04:32:09.940337 16774 raft_consensus.cc:3037] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:09.940337 16387 raft_consensus.cc:3037] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:09.941699 16774 raft_consensus.cc:2454] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57a0bf03f31e4b62979e3591aa6b14cb in term 1.
I20260806 04:32:09.941699 16387 raft_consensus.cc:2454] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57a0bf03f31e4b62979e3591aa6b14cb in term 1.
I20260806 04:32:09.941992 16486 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0; no voters:
I20260806 04:32:09.942250 16863 raft_consensus.cc:2781] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:09.942466 16863 raft_consensus.cc:686] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 LEADER]: Becoming Leader. State: Replica: 57a0bf03f31e4b62979e3591aa6b14cb, State: Running, Role: LEADER
I20260806 04:32:09.942827 16863 consensus_queue.cc:237] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:09.946193 16179 catalog_manager.cc:5495] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2). New cstate: current_term: 1 leader_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } health_report { overall_health: UNKNOWN } } }
I20260806 04:32:09.983065 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:09.979481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.28.94.2:42207)
0806 04:32:09.979640 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:09.979644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:09.979659 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:32:09.979770 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:09.979772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:09.979837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:09.979933 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:09.980218 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.980727 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.981583 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:09.981800 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:09.981890 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:09.982213 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:09.982663 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:09.982773 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:09.982845 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 04:32:09.982908 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:32:09.984495 16843 tablet.cc:2369] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:09.990082 16387 raft_consensus.cc:1261] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Refusing update from remote peer 57a0bf03f31e4b62979e3591aa6b14cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:09.990091 16774 raft_consensus.cc:1261] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Refusing update from remote peer 57a0bf03f31e4b62979e3591aa6b14cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:09.990869 16870 consensus_queue.cc:1035] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:09.991081 16863 consensus_queue.cc:1035] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:09.998878 16875 mvcc.cc:205] Tried to move back new op lower bound from 7315418030030856192 to 7315418029848563712. Current Snapshot: MvccSnapshot[applied={T|T < 7315418030030856192}]
I20260806 04:32:10.008095 16176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:10.008282 16176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:10.010828 16176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Sending DeleteTablet for 3 replicas of tablet bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:10.011642 16355 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc9313fdf604babb8a5197e3ce6443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:10 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:37776
I20260806 04:32:10.011708 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 16235
I20260806 04:32:10.011667 16744 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc9313fdf604babb8a5197e3ce6443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:10 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:56876
I20260806 04:32:10.016258 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc9313fdf604babb8a5197e3ce6443b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:10 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:39422
I20260806 04:32:10.019243 16889 tablet_replica.cc:330] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: stopping tablet replica
I20260806 04:32:10.019676 16889 raft_consensus.cc:2229] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:10.020123 16890 tablet_replica.cc:330] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: stopping tablet replica
I20260806 04:32:10.020143 16889 raft_consensus.cc:2258] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:10.020548 16890 raft_consensus.cc:2229] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:32:10.021070 16890 raft_consensus.cc:2258] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:10.022235 16889 ts_tablet_manager.cc:1905] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:10.023566 16890 ts_tablet_manager.cc:1905] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:10.025703 16889 ts_tablet_manager.cc:1918] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:10.025864 16889 log.cc:1196] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:10.026273 16889 ts_tablet_manager.cc:1939] T bfc9313fdf604babb8a5197e3ce6443b P ef8373ef02764282a7ae04ceae475ea0: Deleting consensus metadata
I20260806 04:32:10.026422 16890 ts_tablet_manager.cc:1918] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:10.026607 16890 log.cc:1196] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:10.026979 16890 ts_tablet_manager.cc:1939] T bfc9313fdf604babb8a5197e3ce6443b P 57a0bf03f31e4b62979e3591aa6b14cb: Deleting consensus metadata
I20260806 04:32:10.027824 16156 catalog_manager.cc:4841] TS ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): tablet bfc9313fdf604babb8a5197e3ce6443b (table test-table [id=64bdccba937a4a2cade1af6b68ca1b44]) successfully deleted
I20260806 04:32:10.028084 16156 catalog_manager.cc:4841] TS 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): tablet bfc9313fdf604babb8a5197e3ce6443b (table test-table [id=64bdccba937a4a2cade1af6b68ca1b44]) successfully deleted
W20260806 04:32:10.030748 16162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:39105 (error 108)
I20260806 04:32:10.031106 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 16454
I20260806 04:32:10.038245 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.037869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39105 (local address 127.0.0.1:37782)
0806 04:32:10.038049 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.038132 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 04:32:10.038441 16162 catalog_manager.cc:4584] TS a420562b4f05446199b5bcf1cdf50149 (127.28.94.1:39105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfc9313fdf604babb8a5197e3ce6443b: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39105: connect: Connection refused (error 111)
I20260806 04:32:10.044971 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 16652
I20260806 04:32:10.054742 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 16130
I20260806 04:32:10.067648 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--webserver_interface=127.28.94.62
--webserver_port=39717
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:10.186364 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:10.186659 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:10.186707 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:10.190798 16893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:10.190871 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:10.190888 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:10.190904 16893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:10.190918 16893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:10.196486 16893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=39717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:10.197713 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:10.198794 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:10.207659 16902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.207831 16893 server_base.cc:1034] running on GCE node
W20260806 04:32:10.207660 16904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.207697 16901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.208570 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:10.209805 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:10.210991 16893 hybrid_clock.cc:648] HybridClock initialized: now 1785990730210960 us; error 43 us; skew 500 ppm
Aug 06 04:32:10 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:10.213936 16893 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:10.214948 16893 webserver.cc:460] Webserver started at http://127.28.94.62:39717/ using document root <none> and password file <none>
I20260806 04:32:10.215508 16893 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:10.215570 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:10.219705 16893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:10.221946 16915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.223261 16893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:32:10.223393 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "fd301212a9a84edf8f8528ec773394d4"
format_stamp: "Formatted at 2026-08-06 04:32:07 on dist-test-slave-ncp9"
server_key: "392a55c5ef47ed0444ad130724a20926"
server_key_iv: "40fd3066abd95bd4d167405fd939cabc"
server_key_version: "encryptionkey@0"
I20260806 04:32:10.223973 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:10.239683 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:10.242947 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:10.243148 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:10.250440 16893 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46715
I20260806 04:32:10.250733 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46715 every 8 connection(s)
I20260806 04:32:10.251731 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:10.255334 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16893
I20260806 04:32:10.255759 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:39105
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=33769
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:10.256503 16992 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:32:10.258397 16992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap starting.
I20260806 04:32:10.283897 16992 log.cc:824] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:10.299177 16992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:10.299692 16992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4: Bootstrap complete.
I20260806 04:32:10.305500 16992 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:10.306048 16992 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Initialized, Role: FOLLOWER
I20260806 04:32:10.306666 16992 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:10.306821 16992 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:10.306891 16992 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:10.306988 16992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:32:10.309690 16992 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:10.310073 16992 leader_election.cc:302] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd301212a9a84edf8f8528ec773394d4; no voters:
I20260806 04:32:10.310475 16992 leader_election.cc:288] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:32:10.310606 16996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:32:10.310894 16996 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [term 3 LEADER]: Becoming Leader. State: Replica: fd301212a9a84edf8f8528ec773394d4, State: Running, Role: LEADER
I20260806 04:32:10.311245 16996 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } }
I20260806 04:32:10.311826 16992 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:10.312773 16998 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:10.312927 16998 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:10.312974 16999 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd301212a9a84edf8f8528ec773394d4. Latest consensus state: current_term: 3 leader_uuid: "fd301212a9a84edf8f8528ec773394d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd301212a9a84edf8f8528ec773394d4" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46715 } } }
I20260806 04:32:10.313081 16999 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd301212a9a84edf8f8528ec773394d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:10.314334 17007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:10.319550 17007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:10.320304 17007 catalog_manager.cc:1259] Loaded cluster ID: 02d97a847d0a40d7bdb3360f66bae0f5
I20260806 04:32:10.320369 17007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 04:32:10.321595 17007 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7fc687b9ad google::LogMessage::Fail() at ??:0
@ 0x7f7fc687e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7fc687b4de google::LogMessage::Flush() at ??:0
@ 0x7f7fc687ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7fd11bafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7fd11bbd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7fd11b8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7fd11fc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7fc8acd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7fc8acdc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7fc8acf3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7fc8abe512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7fca1026db start_thread at ??:0
@ 0x7f7fc537961f clone at ??:0
W20260806 04:32:10.425946 16994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:10.426339 16994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:10.426424 16994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:10.432487 16994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:10.432610 16994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:10.432710 16994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:32:10.440946 16994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:39105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=33769
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:10.442620 16994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:10.444090 16994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:10.456844 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.457146 17051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.456844 17049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.457954 16994 server_base.cc:1034] running on GCE node
I20260806 04:32:10.458539 16994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:10.459434 16994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:10.460665 16994 hybrid_clock.cc:648] HybridClock initialized: now 1785990730460631 us; error 41 us; skew 500 ppm
Aug 06 04:32:10 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:10.474949 16994 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:32:10.476122 16994 webserver.cc:460] Webserver started at http://127.28.94.1:33769/ using document root <none> and password file <none>
I20260806 04:32:10.476760 16994 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:10.476836 16994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:10.481168 16994 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:10.483469 17060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.484779 16994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:10.484915 16994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "a420562b4f05446199b5bcf1cdf50149"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "32825b2fb08f8a7ea6ebcf7a9d4d746e"
server_key_iv: "3feed1bdc47a7955bcaa90535bb0e0d3"
server_key_version: "encryptionkey@0"
I20260806 04:32:10.485491 16994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:10.529168 16994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:10.532461 16994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:10.532641 16994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:10.533354 16994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:10.534552 17070 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:32:10.540045 16994 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:32:10.540125 16994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260806 04:32:10.540203 16994 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:32:10.542001 16994 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:32:10.542049 16994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 04:32:10.542431 17070 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Bootstrap starting.
I20260806 04:32:10.552224 16994 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:39105
I20260806 04:32:10.552269 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:39105 every 8 connection(s)
I20260806 04:32:10.553296 16994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:32:10.557144 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 16994
I20260806 04:32:10.557541 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:42207
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=35971
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:10.566805 17070 log.cc:824] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:10.574328 17070 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:10.574707 17070 tablet_bootstrap.cc:492] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Bootstrap complete.
I20260806 04:32:10.575732 17070 ts_tablet_manager.cc:1397] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.008s
I20260806 04:32:10.578300 17070 raft_consensus.cc:348] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:10.578725 17070 raft_consensus.cc:729] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420562b4f05446199b5bcf1cdf50149, State: Initialized, Role: FOLLOWER
I20260806 04:32:10.579211 17070 consensus_queue.cc:260] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:10.580212 17070 ts_tablet_manager.cc:1428] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260806 04:32:10.624091 17229 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:32:10.555293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:57273)
0806 04:32:10.555789 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.555844 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:10.556669 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:10.623261 (+ 66592us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
W20260806 04:32:10.624624 17227 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46715 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46715: Client connection negotiation failed: client connection to 127.28.94.62:46715: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:32:10.625382 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.625116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:40215)
0806 04:32:10.625250 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.625322 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:32:10.626003 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.625806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:48463)
0806 04:32:10.625909 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.625953 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 04:32:10.626160 17227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:32:10.696149 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:10.696439 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:10.696516 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:10.700611 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:10.700691 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:10.700775 17230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:32:10.706266 17230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:42207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=35971
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:10.707441 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:10.708549 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:10.719288 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.719309 17244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.719388 17230 server_base.cc:1034] running on GCE node
W20260806 04:32:10.719300 17242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.719928 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:10.720618 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:10.721787 17230 hybrid_clock.cc:648] HybridClock initialized: now 1785990730721747 us; error 55 us; skew 500 ppm
Aug 06 04:32:10 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:10.725291 17230 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:32:10.726223 17230 webserver.cc:460] Webserver started at http://127.28.94.2:35971/ using document root <none> and password file <none>
I20260806 04:32:10.726794 17230 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:10.726862 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:10.730942 17230 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:10.733150 17254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.734503 17230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:32:10.734633 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "57a0bf03f31e4b62979e3591aa6b14cb"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "dc32d27d1cfcd497725ec1d6969e0058"
server_key_iv: "888e0af7e1dd08fad72d8aa314789f47"
server_key_version: "encryptionkey@0"
I20260806 04:32:10.735221 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:10.754308 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:10.757808 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:10.758010 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:10.758677 17230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:10.759600 17230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:10.759654 17230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.759712 17230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:10.759737 17230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.768751 17230 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42207
I20260806 04:32:10.768791 17420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42207 every 8 connection(s)
I20260806 04:32:10.769851 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:32:10.773825 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.772058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:59671)
0806 04:32:10.772556 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.773049 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
W20260806 04:32:10.774288 17421 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46715 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46715: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
I20260806 04:32:10.774933 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.774710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:43935)
0806 04:32:10.774821 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.774874 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 04:32:10.775556 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.775355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:50517)
0806 04:32:10.775459 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.775505 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 04:32:10.775722 17421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:32:10.777014 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 17230
I20260806 04:32:10.777410 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:35855
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:46715
--webserver_port=37797
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:10.892637 17425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:10.892931 17425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:10.892976 17425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:10.896922 17425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:10.896983 17425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:10.897044 17425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:32:10.902333 17425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=37797
--tserver_master_addrs=127.28.94.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:10.903425 17425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:10.904465 17425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:10.914836 17435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.914836 17437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:10.914872 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:10.915261 17425 server_base.cc:1034] running on GCE node
I20260806 04:32:10.915758 17425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:10.916443 17425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:10.917620 17425 hybrid_clock.cc:648] HybridClock initialized: now 1785990730917596 us; error 32 us; skew 500 ppm
Aug 06 04:32:10 dist-test-slave-ncp9 krb5kdc[15347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:10.920409 17425 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:32:10.921401 17425 webserver.cc:460] Webserver started at http://127.28.94.3:37797/ using document root <none> and password file <none>
I20260806 04:32:10.921988 17425 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:10.922055 17425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:10.926263 17425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:10.928462 17446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.929870 17425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:10.929987 17425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "ef8373ef02764282a7ae04ceae475ea0"
format_stamp: "Formatted at 2026-08-06 04:32:08 on dist-test-slave-ncp9"
server_key: "94b83d2c9d52a59a08d9a97f6576711e"
server_key_iv: "c944f0042629bf3aa9fde635f48b00bd"
server_key_version: "encryptionkey@0"
I20260806 04:32:10.930570 17425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:10.945748 17425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:10.949158 17425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:10.949350 17425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:10.950016 17425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:10.950966 17425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:10.951017 17425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.951076 17425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:10.951105 17425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:10.960201 17425 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:35855
I20260806 04:32:10.960234 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:35855 every 8 connection(s)
I20260806 04:32:10.961339 17425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:32:10.965233 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 17425
I20260806 04:32:10.965564 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.963717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:36993)
0806 04:32:10.964247 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.964690 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":137,"threads_started":1}
W20260806 04:32:10.966058 17603 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46715 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46715: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
I20260806 04:32:10.966835 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.966578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:39485)
0806 04:32:10.966725 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.966775 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:32:10.967417 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:10.967246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:42017)
0806 04:32:10.967332 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:10.967370 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 04:32:10.967582 17603 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:32:11.627163 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.626709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:43475)
0806 04:32:11.626985 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:11.627041 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260806 04:32:11.776795 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.776353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:56579)
0806 04:32:11.776625 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:11.776687 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260806 04:32:11.779620 17609 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:11.779810 17609 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:11.781083 17609 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
Aug 06 04:32:11 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.3@KRBTEST.COM
Aug 06 04:32:11 dist-test-slave-ncp9 krb5kdc[15347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990730, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.2@KRBTEST.COM
I20260806 04:32:11.795490 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.781547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42207 (local address 127.28.94.1:39375)
0806 04:32:11.782070 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:11.782082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:11.782186 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:11.783497 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:11.783500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:11.783782 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:11.784266 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:11.784281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.785588 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.785591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:11.786556 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:11.786568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.787274 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.787278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:11.787485 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:11.788136 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:11.788156 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:11.790264 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:11.792334 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:11.792337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:11.792338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:11.792779 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:11.793248 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:11.793251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:11.793253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:11.793471 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:11.794296 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:11.794304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:11.794672 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 04:32:11.794896 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:32:11.795342 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":242,"spinlock_wait_cycles":17792,"thread_start_us":72,"threads_started":1}
I20260806 04:32:11.795576 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.781453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:58025 (local address 127.28.94.3:35855)
0806 04:32:11.781806 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:11.781819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:11.781856 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:32:11.782529 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:11.782535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:11.782894 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:11.783147 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:11.784434 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.785469 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.786680 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.787159 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.787322 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:11.790120 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:11.790126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:11.790149 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:11.790182 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:11.792168 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:11.792953 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:11.792968 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:11.792972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:11.793139 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:11.793575 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:11.793578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:11.793580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:11.794167 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:11.794316 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:11.795064 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 04:32:11.795379 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260806 04:32:11.795576 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.781698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:39375 (local address 127.28.94.2:42207)
0806 04:32:11.781958 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:11.781989 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:11.782174 (+ 185us) server_negotiation.cc:407] Connection header received
0806 04:32:11.782887 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:11.782892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:11.783228 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:11.783515 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:11.784424 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.785442 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.786689 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.787152 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.787316 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:11.790459 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:11.790465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:11.790482 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:11.790516 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:11.792242 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:11.792953 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:11.792960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:11.792967 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:11.793139 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:11.793574 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:11.793578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:11.793580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:11.794173 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:11.794316 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:11.795064 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 04:32:11.795379 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260806 04:32:11.795490 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.781285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:35855 (local address 127.28.94.1:58025)
0806 04:32:11.781418 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:11.781434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:11.781545 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:11.783256 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:11.783263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:11.783821 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:11.784267 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:11.784281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.785593 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.785597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:11.786558 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:11.786568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:11.787285 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:11.787289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:11.787488 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:11.788128 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:11.788146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:11.789866 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:11.792292 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:11.792299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:11.792315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:11.792779 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:11.793248 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:11.793251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:11.793254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:11.793471 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:11.794296 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:11.794304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:11.794672 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 04:32:11.794896 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:32:11.795279 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":237,"spinlock_wait_cycles":71296}
W20260806 04:32:11.798336 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
W20260806 04:32:11.798560 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:11.798623 17061 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149; no voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0
I20260806 04:32:11.798902 17609 raft_consensus.cc:2726] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:32:11.968719 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:11.968223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:45571)
0806 04:32:11.968558 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:11.968612 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260806 04:32:12.628592 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:12.628056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:46509)
0806 04:32:12.628389 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:12.628462 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":86,"threads_started":1}
I20260806 04:32:12.778079 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:12.777608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:60405)
0806 04:32:12.777908 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:12.777975 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260806 04:32:12.970221 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:12.969617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:33239)
0806 04:32:12.969937 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:12.970019 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260806 04:32:13.629910 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:13.629465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:43761)
0806 04:32:13.629744 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:13.629805 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260806 04:32:13.779501 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:13.778976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:43163)
0806 04:32:13.779308 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:13.779378 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260806 04:32:13.832805 17624 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:13.833003 17624 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:13.833482 17624 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
W20260806 04:32:13.834465 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
W20260806 04:32:13.834610 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:13.834669 17061 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149; no voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0
I20260806 04:32:13.834846 17624 raft_consensus.cc:2726] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:32:13.971413 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:13.971004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:53445)
0806 04:32:13.971252 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:13.971309 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260806 04:32:14.631296 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:14.630787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:39165)
0806 04:32:14.631123 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:14.631193 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
I20260806 04:32:14.780805 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:14.780354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:43965)
0806 04:32:14.780640 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:14.780701 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260806 04:32:14.972747 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:14.972302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:40145)
0806 04:32:14.972574 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:14.972641 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260806 04:32:15.632995 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:15.632375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:60323)
0806 04:32:15.632754 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:15.632826 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":93,"threads_started":1}
I20260806 04:32:15.782088 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:15.781627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:33787)
0806 04:32:15.781921 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:15.781983 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260806 04:32:15.974064 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:15.973617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:55915)
0806 04:32:15.973890 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:15.973956 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260806 04:32:16.634215 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:16.633810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:36437)
0806 04:32:16.634057 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:16.634112 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260806 04:32:16.783334 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:16.782903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:60467)
0806 04:32:16.783174 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:16.783234 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260806 04:32:16.975330 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:16.974845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:41555)
0806 04:32:16.975153 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:16.975227 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260806 04:32:17.200837 17640 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:17.201050 17640 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:17.201512 17640 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
W20260806 04:32:17.202298 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
W20260806 04:32:17.202423 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:17.202461 17061 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149; no voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0
I20260806 04:32:17.202596 17640 raft_consensus.cc:2726] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:32:17.635396 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:17.634902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:54009)
0806 04:32:17.635220 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:17.635285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260806 04:32:17.784541 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:17.784062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:35479)
0806 04:32:17.784368 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:17.784423 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":87,"threads_started":1}
I20260806 04:32:17.976486 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:17.976039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:54701)
0806 04:32:17.976342 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:17.976395 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":73,"threads_started":1}
I20260806 04:32:18.636502 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:18.636116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:38251)
0806 04:32:18.636347 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:18.636400 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260806 04:32:18.785656 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:18.785227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:56567)
0806 04:32:18.785495 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:18.785557 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260806 04:32:18.977583 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:18.977178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:45253)
0806 04:32:18.977428 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:18.977479 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260806 04:32:19.637660 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:19.637240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:47391)
0806 04:32:19.637498 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:19.637561 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260806 04:32:19.786823 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:19.786400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:51517)
0806 04:32:19.786663 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:19.786721 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260806 04:32:19.978766 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:19.978303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:56031)
0806 04:32:19.978598 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:19.978656 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260806 04:32:20.638875 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:20.638471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:51873)
0806 04:32:20.638724 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:20.638777 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
I20260806 04:32:20.787884 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:20.787479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:41747)
0806 04:32:20.787735 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:20.787789 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260806 04:32:20.979948 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:20.979528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:45751)
0806 04:32:20.979805 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:20.979856 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260806 04:32:21.639994 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:21.639566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:44909)
0806 04:32:21.639837 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:21.639893 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260806 04:32:21.789002 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:21.788549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:47137)
0806 04:32:21.788818 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:21.788871 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260806 04:32:21.981088 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:21.980624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:49909)
0806 04:32:21.980925 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:21.980982 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260806 04:32:22.008723 17663 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:22.008858 17663 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:22.009294 17663 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
W20260806 04:32:22.010003 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
W20260806 04:32:22.010125 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:22.010162 17061 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149; no voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0
I20260806 04:32:22.010300 17663 raft_consensus.cc:2726] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:32:22.641213 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:22.640731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:39971)
0806 04:32:22.641055 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:22.641111 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260806 04:32:22.790046 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:22.789626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:33149)
0806 04:32:22.789895 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:22.789952 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260806 04:32:22.982288 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:22.981809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:36905)
0806 04:32:22.982119 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:22.982176 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":127,"threads_started":1}
I20260806 04:32:23.642323 17669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:23.641869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:55919)
0806 04:32:23.642144 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:23.642215 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260806 04:32:23.791239 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:23.790812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:57053)
0806 04:32:23.791089 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:23.791142 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260806 04:32:23.983556 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:23.983088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:48747)
0806 04:32:23.983379 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:23.983451 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260806 04:32:24.643677 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:24.643205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:35683)
0806 04:32:24.643505 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:24.643572 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260806 04:32:24.792375 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:24.791896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:45161)
0806 04:32:24.792212 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:24.792275 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
I20260806 04:32:24.984768 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:24.984311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:32779)
0806 04:32:24.984609 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:24.984668 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260806 04:32:25.430097 17678 raft_consensus.cc:482] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:25.430261 17678 raft_consensus.cc:504] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef8373ef02764282a7ae04ceae475ea0" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 35855 } } peers { permanent_uuid: "57a0bf03f31e4b62979e3591aa6b14cb" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42207 } } peers { permanent_uuid: "a420562b4f05446199b5bcf1cdf50149" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39105 } }
I20260806 04:32:25.430711 17678 leader_election.cc:288] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855), 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207)
W20260806 04:32:25.431540 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 57a0bf03f31e4b62979e3591aa6b14cb (127.28.94.2:42207): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
W20260806 04:32:25.431663 17061 leader_election.cc:341] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef8373ef02764282a7ae04ceae475ea0 (127.28.94.3:35855): Not found: Tablet not found: bfc9313fdf604babb8a5197e3ce6443b
I20260806 04:32:25.431700 17061 leader_election.cc:302] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a420562b4f05446199b5bcf1cdf50149; no voters: 57a0bf03f31e4b62979e3591aa6b14cb, ef8373ef02764282a7ae04ceae475ea0
I20260806 04:32:25.431830 17678 raft_consensus.cc:2726] T bfc9313fdf604babb8a5197e3ce6443b P a420562b4f05446199b5bcf1cdf50149 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:32:25.645081 17680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:25.644581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.1:42861)
0806 04:32:25.644849 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:25.644943 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260806 04:32:25.793709 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:25.793269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.2:58191)
0806 04:32:25.793539 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:25.793609 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260806 04:32:25.967654 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 16994
I20260806 04:32:25.986188 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:25.985741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46715 (local address 127.28.94.3:52737)
0806 04:32:25.986027 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:25.986088 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260806 04:32:25.990378 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 17230
I20260806 04:32:26.002951 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 17425
2026-08-06T04:32:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19560 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:32:26 dist-test-slave-ncp9 krb5kdc[17690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:32:26 dist-test-slave-ncp9 krb5kdc[17690](info): set up 2 sockets
Aug 06 04:32:26 dist-test-slave-ncp9 krb5kdc[17690](info): commencing operation
krb5kdc: starting...
W20260806 04:32:27.113097 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:32:27.232131 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:27.344092 17707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:27.344321 17707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:27.344358 17707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:27.348209 17707 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:32:27.348254 17707 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:32:27.348268 17707 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:32:27.348280 17707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:27.348296 17707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:27.348315 17707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:27.348335 17707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:27.348359 17707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:27.353399 17707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:27.354318 17707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:27.355253 17707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:27.363675 17713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:27.363729 17714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:27.363758 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:27.363812 17707 server_base.cc:1034] running on GCE node
I20260806 04:32:27.364410 17707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:27.365504 17707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:27.366675 17707 hybrid_clock.cc:648] HybridClock initialized: now 1785990747366662 us; error 24 us; skew 500 ppm
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:27.369230 17707 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:27.370065 17707 webserver.cc:460] Webserver started at http://127.28.94.62:40013/ using document root <none> and password file <none>
I20260806 04:32:27.370572 17707 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:27.370628 17707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:27.370797 17707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:27.372725 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "19a05c279a5b444e8f06e75dba5380d7"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "2f74b4d005a3b989c0681523a58b2e00"
server_key_iv: "fc5b908121314d917894e8f0463d9a4c"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.373272 17707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "19a05c279a5b444e8f06e75dba5380d7"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "2f74b4d005a3b989c0681523a58b2e00"
server_key_iv: "fc5b908121314d917894e8f0463d9a4c"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.376735 17707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:27.379033 17729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.380292 17707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:27.380393 17707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "19a05c279a5b444e8f06e75dba5380d7"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "2f74b4d005a3b989c0681523a58b2e00"
server_key_iv: "fc5b908121314d917894e8f0463d9a4c"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.380496 17707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:27.397658 17707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:27.400390 17707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:27.400565 17707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:27.407425 17707 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46167
I20260806 04:32:27.407701 17804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46167 every 8 connection(s)
I20260806 04:32:27.408351 17707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:27.411140 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:27.416601 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap starting.
I20260806 04:32:27.418180 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 17707
I20260806 04:32:27.418299 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:32:27.418624 29048 external_mini_cluster.cc:1428] Setting key 055e9efa2f8993a3ea423f098fa1042a
I20260806 04:32:27.419395 17805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:27.420296 17805 log.cc:824] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:27.422492 17805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: No bootstrap required, opened a new log
I20260806 04:32:27.426600 17805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:27.426872 17805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:27.426960 17805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Initialized, Role: FOLLOWER
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:27.427407 17805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:27.427561 17805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:27.427625 17805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:27.427719 17805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:27.428781 17805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:27.429137 17805 leader_election.cc:302] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a05c279a5b444e8f06e75dba5380d7; no voters:
I20260806 04:32:27.429421 17805 leader_election.cc:288] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:32:27.429564 17812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:27.429831 17812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 LEADER]: Becoming Leader. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Running, Role: LEADER
I20260806 04:32:27.430197 17812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:27.430577 17805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:27.431888 17813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:32:27.432011 17813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:27.432411 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a05c279a5b444e8f06e75dba5380d7. Latest consensus state: current_term: 1 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:32:27.432587 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:27.433676 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:27.419742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.28.94.62:46167)
0806 04:32:27.420122 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:27.420137 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:27.420173 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:32:27.420868 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:27.420921 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:27.421297 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:27.421576 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:27.422499 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.423501 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.424668 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.425182 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.425296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:27.428013 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:27.428019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:27.428036 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:27.428066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:27.429801 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.430326 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.430331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.430336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.430474 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.430809 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.430813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.430815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.431315 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:27.432216 (+ 901us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:27.432373 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 04:32:27.432680 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
W20260806 04:32:27.435774 17832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:32:27.435900 17832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:32:27.436033 17824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:27.436729 17824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:27.442039 17824 catalog_manager.cc:1347] Generated new cluster ID: b2750799c07542da9cd96b56244a1bb2
I20260806 04:32:27.442099 17824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:27.464825 17824 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:32:27.466105 17824 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:27.481516 17824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Generated new TSK 0
I20260806 04:32:27.482331 17824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:32:27.512101 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:27.622354 17843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:27.622601 17843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:27.622658 17843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:27.626600 17843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:27.626655 17843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:27.626731 17843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:32:27.631853 17843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:27.632864 17843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:27.633862 17843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:27.643148 17850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:27.643154 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:27.643323 17843 server_base.cc:1034] running on GCE node
W20260806 04:32:27.643158 17849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:27.643839 17843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:27.644421 17843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:27.645566 17843 hybrid_clock.cc:648] HybridClock initialized: now 1785990747645542 us; error 39 us; skew 500 ppm
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:27.648102 17843 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:32:27.648937 17843 webserver.cc:460] Webserver started at http://127.28.94.1:37443/ using document root <none> and password file <none>
I20260806 04:32:27.649464 17843 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:27.649523 17843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:27.649688 17843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:27.651618 17843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "2b5930fa863a4870b7cd8f43dc80386a"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "1fafe7fa2c9237be1b92972a8ee6f411"
server_key_iv: "d8e2dadce4c1c085764e589d3f0afacd"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.652138 17843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "2b5930fa863a4870b7cd8f43dc80386a"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "1fafe7fa2c9237be1b92972a8ee6f411"
server_key_iv: "d8e2dadce4c1c085764e589d3f0afacd"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.655658 17843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 04:32:27.657954 17865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.659088 17843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:32:27.659200 17843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "2b5930fa863a4870b7cd8f43dc80386a"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "1fafe7fa2c9237be1b92972a8ee6f411"
server_key_iv: "d8e2dadce4c1c085764e589d3f0afacd"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.659296 17843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:27.691565 17843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:27.694649 17843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:27.694821 17843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:27.695420 17843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:27.696349 17843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:27.696396 17843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.696439 17843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:27.696470 17843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.704694 17843 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44193
I20260806 04:32:27.704761 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44193 every 8 connection(s)
I20260806 04:32:27.705612 17843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:32:27.708418 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 17843
I20260806 04:32:27.708554 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:32:27.708843 29048 external_mini_cluster.cc:1428] Setting key 3585cdd006b81d9431b8bd00a4ccde3b
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:27.719102 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:27.707302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:36945 (local address 127.28.94.62:46167)
0806 04:32:27.707468 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:27.707472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:27.708469 (+ 997us) server_negotiation.cc:407] Connection header received
0806 04:32:27.709472 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:27.709475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:27.709536 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:27.709616 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:27.711017 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.711551 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.712931 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.713266 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.713370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:27.715614 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:27.715618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:27.715621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:27.715650 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:27.717155 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.717620 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.717623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.717626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.717715 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.717995 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.717998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.718000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.718238 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:27.718338 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:27.718805 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 04:32:27.718930 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:32:27.719846 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:27.707670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:36945)
0806 04:32:27.708325 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:27.708365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:27.709221 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:27.709769 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:27.709775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:27.710182 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:27.710824 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:27.710845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.711677 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.711681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:27.712794 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:27.712805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.713396 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.713398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:27.713544 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:27.714069 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:27.714085 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:27.715443 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:27.717263 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:27.717267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:27.717279 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:27.717524 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:27.717795 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:27.717797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:27.717799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:27.717915 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:27.718328 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:27.718332 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:27.718576 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 04:32:27.718722 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 04:32:27.718952 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":119,"threads_started":1}
I20260806 04:32:27.720991 18039 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:27.721218 18039 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:27.721717 18039 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:32:27.723223 17756 ts_manager.cc:194] Registered new tserver with Master: 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193)
I20260806 04:32:27.724350 17756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:36945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:32:27.737177 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:27.847411 18047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:27.847636 18047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:27.847675 18047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:27.851501 18047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:27.851544 18047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:27.851596 18047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:32:27.856490 18047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:27.857371 18047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:27.858268 18047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:27.867318 18056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:27.867365 18057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:27.867484 18047 server_base.cc:1034] running on GCE node
W20260806 04:32:27.867689 18059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:27.868057 18047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:27.868618 18047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:27.869779 18047 hybrid_clock.cc:648] HybridClock initialized: now 1785990747869764 us; error 25 us; skew 500 ppm
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:27.872310 18047 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:32:27.873102 18047 webserver.cc:460] Webserver started at http://127.28.94.2:37087/ using document root <none> and password file <none>
I20260806 04:32:27.873586 18047 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:27.873641 18047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:27.873803 18047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:27.875622 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "65a98e145157184daf3665a7901532ee"
server_key_iv: "931555c2f754848be4e4c28880c8f541"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.876113 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "65a98e145157184daf3665a7901532ee"
server_key_iv: "931555c2f754848be4e4c28880c8f541"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.879523 18047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260806 04:32:27.881908 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.883121 18047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 04:32:27.883216 18047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "65a98e145157184daf3665a7901532ee"
server_key_iv: "931555c2f754848be4e4c28880c8f541"
server_key_version: "encryptionkey@0"
I20260806 04:32:27.883301 18047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:27.893990 18047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:27.896818 18047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:27.897001 18047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:27.897549 18047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:27.898294 18047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:27.898339 18047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.898380 18047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:27.898408 18047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:27.906319 18047 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:40863
I20260806 04:32:27.906375 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:40863 every 8 connection(s)
I20260806 04:32:27.907123 18047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:32:27.912820 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 18047
I20260806 04:32:27.912945 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:32:27.913228 29048 external_mini_cluster.cc:1428] Setting key 4f83a43e7b7d3267851c4f8dba3f18c4
Aug 06 04:32:27 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990747, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:27.919339 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:27.908806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:53033 (local address 127.28.94.62:46167)
0806 04:32:27.908970 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:27.908984 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:27.909967 (+ 983us) server_negotiation.cc:407] Connection header received
0806 04:32:27.910751 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:27.910754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:27.910799 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:27.910866 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:27.911909 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.912431 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.913616 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.913840 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.913927 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:27.916152 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:27.916155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:27.916157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:27.916180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:27.917499 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.917950 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.917954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.917955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.918032 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:27.918329 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:27.918333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:27.918335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:27.918521 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:27.918605 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:27.919058 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 04:32:27.919181 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:32:27.920027 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:27.909188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:53033)
0806 04:32:27.909832 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:27.909871 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:27.910592 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:27.911004 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:27.911010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:27.911332 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:27.911794 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:27.911808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.912512 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.912514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:27.913483 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:27.913493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:27.913984 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:27.913988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:27.914142 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:27.914644 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:27.914659 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:27.915992 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:27.917608 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:27.917614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:27.917627 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:27.917858 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:27.918114 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:27.918117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:27.918119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:27.918243 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:27.918618 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:27.918625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:27.918839 (+ 214us) client_negotiation.cc:769] Sending connection context
0806 04:32:27.919002 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 04:32:27.919294 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":115,"threads_started":1}
I20260806 04:32:27.921222 18226 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:27.921443 18226 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:27.921890 18226 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
I20260806 04:32:27.922899 17756 ts_manager.cc:194] Registered new tserver with Master: 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863)
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:32:27.923434 17756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:53033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:32:27.940454 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:28.052206 18234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:28.052428 18234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:28.052475 18234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:28.056361 18234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:28.056411 18234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:28.056475 18234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:32:28.061445 18234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:28.062414 18234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:28.063406 18234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:28.072619 18245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:28.072628 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:28.072655 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:28.072788 18234 server_base.cc:1034] running on GCE node
I20260806 04:32:28.073210 18234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:28.073822 18234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:28.074975 18234 hybrid_clock.cc:648] HybridClock initialized: now 1785990748074963 us; error 31 us; skew 500 ppm
Aug 06 04:32:28 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990748, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:28.077523 18234 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:32:28.078328 18234 webserver.cc:460] Webserver started at http://127.28.94.3:46313/ using document root <none> and password file <none>
I20260806 04:32:28.078847 18234 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:28.078907 18234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:28.079110 18234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:32:28.081041 18234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "fdf37a8e765342a5b5a1f07712f9218b"
format_stamp: "Formatted at 2026-08-06 04:32:28 on dist-test-slave-ncp9"
server_key: "97e38ce4db078a446d4c21efdfda153c"
server_key_iv: "a218385951921d5964b519edff906ed5"
server_key_version: "encryptionkey@0"
I20260806 04:32:28.081561 18234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "fdf37a8e765342a5b5a1f07712f9218b"
format_stamp: "Formatted at 2026-08-06 04:32:28 on dist-test-slave-ncp9"
server_key: "97e38ce4db078a446d4c21efdfda153c"
server_key_iv: "a218385951921d5964b519edff906ed5"
server_key_version: "encryptionkey@0"
I20260806 04:32:28.085157 18234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:28.087507 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:28.088672 18234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:32:28.088780 18234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "fdf37a8e765342a5b5a1f07712f9218b"
format_stamp: "Formatted at 2026-08-06 04:32:28 on dist-test-slave-ncp9"
server_key: "97e38ce4db078a446d4c21efdfda153c"
server_key_iv: "a218385951921d5964b519edff906ed5"
server_key_version: "encryptionkey@0"
I20260806 04:32:28.088871 18234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:28.103289 18234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:28.105774 18234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:28.105938 18234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:28.106479 18234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:28.107256 18234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:28.107313 18234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:28.107364 18234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:28.107393 18234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:28.115526 18234 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:32941
I20260806 04:32:28.115602 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:32941 every 8 connection(s)
I20260806 04:32:28.116355 18234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:32:28.117048 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 18234
I20260806 04:32:28.117157 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:32:28.117463 29048 external_mini_cluster.cc:1428] Setting key bdc9a6cef12da06e47660bc5f5f03f16
Aug 06 04:32:28 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990748, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:28.128365 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.118139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:33431 (local address 127.28.94.62:46167)
0806 04:32:28.118258 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.118263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.119451 (+ 1188us) server_negotiation.cc:407] Connection header received
0806 04:32:28.120328 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.120331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.120377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.120439 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:28.121546 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.122097 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.123135 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.123335 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.123401 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.125527 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:28.125530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:28.125532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:28.125557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:28.126847 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:28.127248 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:28.127250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:28.127251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:28.127318 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:28.127554 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:28.127556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:28.127557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:28.127708 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:28.127775 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.128162 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.128250 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:32:28.129128 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.118553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:33431)
0806 04:32:28.119289 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.119343 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.120156 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.120566 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.120572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.120956 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:28.121429 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.121443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.122185 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.122197 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.123046 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.123055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.123433 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.123435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.123565 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.124019 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:28.124034 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:28.125379 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:28.126932 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:28.126936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:28.126945 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:28.127178 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:28.127377 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:28.127378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:28.127380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:28.127493 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:28.127775 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:28.127779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:28.127979 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.128132 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.128351 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":129,"threads_started":1}
I20260806 04:32:28.130235 18417 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:28.130456 18417 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:28.130951 18417 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
I20260806 04:32:28.131948 17756 ts_manager.cc:194] Registered new tserver with Master: fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
I20260806 04:32:28.132536 17756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:33431
I20260806 04:32:28.142634 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:32:28.150954 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.144475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.28.94.62:46167)
0806 04:32:28.144627 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.144632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.144739 (+ 107us) server_negotiation.cc:407] Connection header received
0806 04:32:28.144830 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.144832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.144867 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.144950 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:28.145609 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.146089 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.146900 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.147112 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.147196 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.148379 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:28.148381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:28.148383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:28.148405 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:28.149455 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:28.149837 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:28.149840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:28.149841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:28.149913 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:28.150157 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:28.150161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:28.150162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:28.150357 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:28.150475 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.150711 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.150815 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:32:28.153475 17756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:32:28.156325 17756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:32:28.170096 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.164610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44193 (local address 127.0.0.1:58850)
0806 04:32:28.164991 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.165041 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.165186 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.165927 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.165934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.165957 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:28.166116 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.166124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.167268 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.167271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.168467 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.168477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.169544 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.169548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.169658 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.169703 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.169820 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.169940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260806 04:32:28.170104 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.164684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.28.94.1:44193)
0806 04:32:28.165024 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.165041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.165114 (+ 73us) server_negotiation.cc:407] Connection header received
0806 04:32:28.165360 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.165366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.165573 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.165729 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:28.166245 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.167152 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.168587 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.169422 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.169531 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.169681 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.169855 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.169956 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":125,"threads_started":1}
I20260806 04:32:28.170810 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.165344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:32941 (local address 127.0.0.1:59476)
0806 04:32:28.165836 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.165850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.165941 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.166645 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.166649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.166669 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:28.166767 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.166774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.168150 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.168154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.169119 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.169130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.170386 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.170390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.170479 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.170507 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.170615 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.170690 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":90,"threads_started":1}
I20260806 04:32:28.170929 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.165548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.28.94.3:32941)
0806 04:32:28.166285 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.166295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.166312 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:32:28.166372 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.166375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.166517 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.166622 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:28.166896 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.167981 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.169234 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.169920 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.170017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.170150 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.170665 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.170764 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":156,"threads_started":1}
I20260806 04:32:28.171176 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.165645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:40863 (local address 127.0.0.1:60040)
0806 04:32:28.165966 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.165979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.166058 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.167315 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.167318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.167339 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:28.167417 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.167422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.168687 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.168691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.169783 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.169794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.170814 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.170818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.170905 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.170930 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.170981 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.171049 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
I20260806 04:32:28.171753 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.166069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.28.94.2:40863)
0806 04:32:28.166875 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.166887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.166907 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:32:28.166978 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.166983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.167152 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.167339 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:28.167508 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.168565 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.169922 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.170685 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.171334 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.171479 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.171529 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.171617 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":113,"threads_started":1}
I20260806 04:32:28.173614 17938 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a (DEFAULT_TABLE table=test-table [id=024f267c36bc4ecdaeddd9ec924fc112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:28.173924 18318 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a (DEFAULT_TABLE table=test-table [id=024f267c36bc4ecdaeddd9ec924fc112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:28.173961 18129 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a (DEFAULT_TABLE table=test-table [id=024f267c36bc4ecdaeddd9ec924fc112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:28.174820 17938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9a8177f90f466da1e8abf2a790c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:28.174830 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9a8177f90f466da1e8abf2a790c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:28.174834 18129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9a8177f90f466da1e8abf2a790c28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:28.181005 18438 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Bootstrap starting.
I20260806 04:32:28.182578 18441 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Bootstrap starting.
I20260806 04:32:28.182578 18439 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap starting.
I20260806 04:32:28.183876 18438 tablet_bootstrap.cc:654] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:28.184679 18438 log.cc:824] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:28.184707 18439 tablet_bootstrap.cc:654] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:28.185164 18441 tablet_bootstrap.cc:654] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:28.185956 18439 log.cc:824] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:28.185957 18441 log.cc:824] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:28.187016 18438 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: No bootstrap required, opened a new log
I20260806 04:32:28.187206 18438 ts_tablet_manager.cc:1397] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 04:32:28.187935 18441 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: No bootstrap required, opened a new log
I20260806 04:32:28.187935 18439 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: No bootstrap required, opened a new log
I20260806 04:32:28.188122 18441 ts_tablet_manager.cc:1397] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:32:28.188122 18439 ts_tablet_manager.cc:1397] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:32:28.190109 18438 raft_consensus.cc:348] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.190326 18438 raft_consensus.cc:374] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:28.190397 18438 raft_consensus.cc:729] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7c5f3ee3f4493a24fad90d9a991c4, State: Initialized, Role: FOLLOWER
I20260806 04:32:28.190788 18438 consensus_queue.cc:260] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.191434 18226 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
I20260806 04:32:28.191668 18438 ts_tablet_manager.cc:1428] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260806 04:32:28.191812 18439 raft_consensus.cc:348] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.192118 18439 raft_consensus.cc:374] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:28.192210 18439 raft_consensus.cc:729] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5930fa863a4870b7cd8f43dc80386a, State: Initialized, Role: FOLLOWER
I20260806 04:32:28.192746 18439 consensus_queue.cc:260] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.192744 18441 raft_consensus.cc:348] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.193041 18441 raft_consensus.cc:374] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:28.193101 18441 raft_consensus.cc:729] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf37a8e765342a5b5a1f07712f9218b, State: Initialized, Role: FOLLOWER
I20260806 04:32:28.193471 18441 consensus_queue.cc:260] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.193686 18439 ts_tablet_manager.cc:1428] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260806 04:32:28.194010 18039 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
I20260806 04:32:28.194087 18417 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
I20260806 04:32:28.194392 18441 ts_tablet_manager.cc:1428] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 04:32:28.206651 18040 tablet.cc:2369] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:28.225687 18448 raft_consensus.cc:482] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:28.225833 18448 raft_consensus.cc:504] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.226832 18448 leader_election.cc:288] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941), 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863)
I20260806 04:32:28.231293 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.227041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:32941 (local address 127.28.94.1:52687)
0806 04:32:28.227273 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.227288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.227386 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.227653 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.227656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.227676 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:28.227807 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.227814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.228763 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.228765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.230063 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.230074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.230953 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.230956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.231035 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.231058 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.231119 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.231175 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":30,"thread_start_us":78,"threads_started":1}
I20260806 04:32:28.231375 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.227125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:52687 (local address 127.28.94.3:32941)
0806 04:32:28.227242 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.227245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.227431 (+ 186us) server_negotiation.cc:407] Connection header received
0806 04:32:28.227492 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.227495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.227543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.227616 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:28.227905 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.228653 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.230196 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.230844 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.230955 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.231017 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.231160 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.231232 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:32:28.231575 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.227390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:40863 (local address 127.28.94.1:59091)
0806 04:32:28.227672 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.227684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:28.227778 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:28.227997 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:28.228000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:28.228016 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:28.228106 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.228111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.228972 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.228975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.230111 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:28.230118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.231116 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.231120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:28.231222 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.231246 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:32:28.231322 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:32:28.231438 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":12,"thread_start_us":107,"threads_started":1}
I20260806 04:32:28.231576 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.227499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:59091 (local address 127.28.94.2:40863)
0806 04:32:28.227641 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.227645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.227755 (+ 110us) server_negotiation.cc:407] Connection header received
0806 04:32:28.227856 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.227859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.227907 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.227984 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:28.228197 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.228859 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.230228 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.230838 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.230955 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.231017 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.231388 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.231464 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:32:28.232249 18350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9a8177f90f466da1e8abf2a790c28a" candidate_uuid: "2b5930fa863a4870b7cd8f43dc80386a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf37a8e765342a5b5a1f07712f9218b" is_pre_election: true
I20260806 04:32:28.232247 18156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9a8177f90f466da1e8abf2a790c28a" candidate_uuid: "2b5930fa863a4870b7cd8f43dc80386a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" is_pre_election: true
I20260806 04:32:28.232506 18350 raft_consensus.cc:2454] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5930fa863a4870b7cd8f43dc80386a in term 0.
I20260806 04:32:28.232524 18156 raft_consensus.cc:2454] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5930fa863a4870b7cd8f43dc80386a in term 0.
I20260806 04:32:28.232991 17868 leader_election.cc:302] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a, 2cb7c5f3ee3f4493a24fad90d9a991c4; no voters:
I20260806 04:32:28.233242 18448 raft_consensus.cc:2781] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:28.233332 18448 raft_consensus.cc:482] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:28.233382 18448 raft_consensus.cc:3037] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:28.234370 18448 raft_consensus.cc:504] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.234665 18448 leader_election.cc:288] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 1 election: Requested vote from peers fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941), 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863)
I20260806 04:32:28.234987 18350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9a8177f90f466da1e8abf2a790c28a" candidate_uuid: "2b5930fa863a4870b7cd8f43dc80386a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf37a8e765342a5b5a1f07712f9218b"
I20260806 04:32:28.235054 18156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9a8177f90f466da1e8abf2a790c28a" candidate_uuid: "2b5930fa863a4870b7cd8f43dc80386a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
I20260806 04:32:28.235123 18350 raft_consensus.cc:3037] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:28.235146 18156 raft_consensus.cc:3037] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:28.236349 18350 raft_consensus.cc:2454] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5930fa863a4870b7cd8f43dc80386a in term 1.
I20260806 04:32:28.236349 18156 raft_consensus.cc:2454] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5930fa863a4870b7cd8f43dc80386a in term 1.
I20260806 04:32:28.236632 17868 leader_election.cc:302] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a, fdf37a8e765342a5b5a1f07712f9218b; no voters:
I20260806 04:32:28.236788 18448 raft_consensus.cc:2781] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:28.236999 18448 raft_consensus.cc:686] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 LEADER]: Becoming Leader. State: Replica: 2b5930fa863a4870b7cd8f43dc80386a, State: Running, Role: LEADER
I20260806 04:32:28.237299 18448 consensus_queue.cc:237] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:28.240295 17756 catalog_manager.cc:5495] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1). New cstate: current_term: 1 leader_uuid: "2b5930fa863a4870b7cd8f43dc80386a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260806 04:32:28.265014 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.261884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.28.94.1:44193)
0806 04:32:28.262019 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:28.262022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:28.262035 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:32:28.262104 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:28.262106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:28.262157 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:28.262261 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:28.262423 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.262932 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.263733 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:28.263939 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:28.264020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:28.264292 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:28.264708 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:28.264800 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:28.264826 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 04:32:28.264874 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:32:28.270821 18156 raft_consensus.cc:1261] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Refusing update from remote peer 2b5930fa863a4870b7cd8f43dc80386a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:28.270874 18350 raft_consensus.cc:1261] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Refusing update from remote peer 2b5930fa863a4870b7cd8f43dc80386a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:28.271495 18448 consensus_queue.cc:1035] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:28.271699 18452 consensus_queue.cc:1035] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:28.279279 18457 mvcc.cc:205] Tried to move back new op lower bound from 7315418104909742080 to 7315418104782512128. Current Snapshot: MvccSnapshot[applied={T|T < 7315418104909742080}]
I20260806 04:32:28.281706 18460 mvcc.cc:205] Tried to move back new op lower bound from 7315418104909742080 to 7315418104782512128. Current Snapshot: MvccSnapshot[applied={T|T < 7315418104909742080}]
I20260806 04:32:28.286978 17756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:28.287156 17756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:28.289654 17756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Sending DeleteTablet for 3 replicas of tablet 4d9a8177f90f466da1e8abf2a790c28a
I20260806 04:32:28.290354 17938 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:58850
I20260806 04:32:28.290428 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 17843
I20260806 04:32:28.290527 18318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260806 04:32:28.296074 18129 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60040
I20260806 04:32:28.296185 18474 tablet_replica.cc:330] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: stopping tablet replica
I20260806 04:32:28.296536 18474 raft_consensus.cc:2229] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:28.296583 18475 tablet_replica.cc:330] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: stopping tablet replica
I20260806 04:32:28.296872 18474 raft_consensus.cc:2258] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:28.296942 18475 raft_consensus.cc:2229] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:28.297245 18475 raft_consensus.cc:2258] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:28.298249 18474 ts_tablet_manager.cc:1905] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:28.298750 18475 ts_tablet_manager.cc:1905] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:28.300426 18474 ts_tablet_manager.cc:1918] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:28.300514 18474 log.cc:1196] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/4d9a8177f90f466da1e8abf2a790c28a
I20260806 04:32:28.300808 18474 ts_tablet_manager.cc:1939] T 4d9a8177f90f466da1e8abf2a790c28a P fdf37a8e765342a5b5a1f07712f9218b: Deleting consensus metadata
I20260806 04:32:28.301555 18475 ts_tablet_manager.cc:1918] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:28.301658 18475 log.cc:1196] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/4d9a8177f90f466da1e8abf2a790c28a
I20260806 04:32:28.301856 17734 catalog_manager.cc:4841] TS fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): tablet 4d9a8177f90f466da1e8abf2a790c28a (table test-table [id=024f267c36bc4ecdaeddd9ec924fc112]) successfully deleted
I20260806 04:32:28.302027 18475 ts_tablet_manager.cc:1939] T 4d9a8177f90f466da1e8abf2a790c28a P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting consensus metadata
I20260806 04:32:28.303160 17734 catalog_manager.cc:4841] TS 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): tablet 4d9a8177f90f466da1e8abf2a790c28a (table test-table [id=024f267c36bc4ecdaeddd9ec924fc112]) successfully deleted
W20260806 04:32:28.304265 17734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:44193 (error 108)
I20260806 04:32:28.305049 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 18047
I20260806 04:32:28.305294 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:28.305012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44193 (local address 127.0.0.1:58858)
0806 04:32:28.305145 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:28.305215 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:32:28.311363 17734 catalog_manager.cc:4584] TS 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d9a8177f90f466da1e8abf2a790c28a: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44193: connect: Connection refused (error 111)
I20260806 04:32:28.316795 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 18234
I20260806 04:32:28.325788 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 17707
I20260806 04:32:48.335100 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--webserver_interface=127.28.94.62
--webserver_port=40013
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:32:48.458892 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:48.459228 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:48.459271 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:48.463621 18477 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:32:48.463701 18477 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:32:48.463717 18477 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:32:48.463729 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:32:48.463747 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:48.463764 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:48.463785 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:32:48.463815 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:32:48.469704 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:48.470995 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:48.472152 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:48.487429 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:48.487471 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:48.487507 18477 server_base.cc:1034] running on GCE node
W20260806 04:32:48.487677 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:48.488343 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:48.489763 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:48.490957 18477 hybrid_clock.cc:648] HybridClock initialized: now 1785990768490926 us; error 48 us; skew 500 ppm
Aug 06 04:32:48 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:48.494076 18477 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:32:48.495190 18477 webserver.cc:460] Webserver started at http://127.28.94.62:40013/ using document root <none> and password file <none>
I20260806 04:32:48.495841 18477 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:48.495940 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:48.500590 18477 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:32:48.503317 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:48.504798 18477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:32:48.505005 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "19a05c279a5b444e8f06e75dba5380d7"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "2f74b4d005a3b989c0681523a58b2e00"
server_key_iv: "fc5b908121314d917894e8f0463d9a4c"
server_key_version: "encryptionkey@0"
I20260806 04:32:48.505693 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:48.521018 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:48.524362 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:48.524605 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:48.532573 18477 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46167
I20260806 04:32:48.533031 18582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46167 every 8 connection(s)
I20260806 04:32:48.533902 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:32:48.538909 18583 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:32:48.540957 18583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap starting.
I20260806 04:32:48.544301 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 18477
I20260806 04:32:48.544744 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:44193
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=37443
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:48.560411 18583 log.cc:824] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:48.570401 18583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:48.570815 18583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap complete.
I20260806 04:32:48.575237 18583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:48.575717 18583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Initialized, Role: FOLLOWER
I20260806 04:32:48.576299 18583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:48.576442 18583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:32:48.576524 18583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:32:48.576627 18583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:32:48.578922 18583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:48.579322 18583 leader_election.cc:302] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a05c279a5b444e8f06e75dba5380d7; no voters:
I20260806 04:32:48.579698 18583 leader_election.cc:288] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:32:48.579828 18588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:32:48.580054 18588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 LEADER]: Becoming Leader. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Running, Role: LEADER
I20260806 04:32:48.580469 18588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:32:48.581249 18583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:32:48.582013 18592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a05c279a5b444e8f06e75dba5380d7. Latest consensus state: current_term: 2 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:32:48.582154 18592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:48.582942 18591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:32:48.583035 18591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:32:48.583671 18598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:32:48.589484 18598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:32:48.590307 18598 catalog_manager.cc:1259] Loaded cluster ID: b2750799c07542da9cd96b56244a1bb2
I20260806 04:32:48.590380 18598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:32:48.592942 18598 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:32:48.593793 18598 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Loaded TSK: 0
I20260806 04:32:48.611241 18598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Generated new TSK 1
I20260806 04:32:48.611593 18598 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:32:48.677727 18585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:48.678040 18585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:48.678086 18585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:48.682706 18585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:48.682827 18585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:48.682925 18585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:32:48.688952 18585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:44193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=37443
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:48.690174 18585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:48.691300 18585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:48.703025 18626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:48.703025 18624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:48.703069 18623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:48.703799 18585 server_base.cc:1034] running on GCE node
I20260806 04:32:48.704310 18585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:48.705111 18585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:48.706316 18585 hybrid_clock.cc:648] HybridClock initialized: now 1785990768706288 us; error 38 us; skew 500 ppm
Aug 06 04:32:48 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:48.709755 18585 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:32:48.710884 18585 webserver.cc:460] Webserver started at http://127.28.94.1:37443/ using document root <none> and password file <none>
I20260806 04:32:48.711519 18585 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:48.711627 18585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:48.716147 18585 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:48.718685 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:48.720120 18585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:48.720256 18585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "2b5930fa863a4870b7cd8f43dc80386a"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "1fafe7fa2c9237be1b92972a8ee6f411"
server_key_iv: "d8e2dadce4c1c085764e589d3f0afacd"
server_key_version: "encryptionkey@0"
I20260806 04:32:48.720886 18585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:48.742633 18585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:48.746718 18585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:48.746934 18585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:48.747642 18585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:48.748868 18645 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:32:48.753486 18585 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:32:48.753592 18585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 04:32:48.753671 18585 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:32:48.755628 18585 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:32:48.755690 18585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:32:48.756004 18645 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap starting.
I20260806 04:32:48.766304 18585 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44193
I20260806 04:32:48.766435 18796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44193 every 8 connection(s)
I20260806 04:32:48.767745 18585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:32:48.773814 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 18585
I20260806 04:32:48.774360 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:40863
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=37087
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:32:48.775954 18645 log.cc:824] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:32:48 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:48.785975 18645 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:32:48.786448 18645 tablet_bootstrap.cc:492] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap complete.
I20260806 04:32:48.787783 18645 ts_tablet_manager.cc:1397] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260806 04:32:48.789124 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:48.769991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:59949)
0806 04:32:48.770768 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:48.770827 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:48.771681 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:48.773233 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:48.773242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:48.773779 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:48.774520 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:48.774548 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.776534 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.776538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:48.777944 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:48.777960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.779030 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.779034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:48.779254 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:48.780043 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:48.780069 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:48.782087 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:48.785069 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:48.785077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:48.785091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:48.785484 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:48.785962 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:48.785965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:48.785967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:48.786194 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:48.787051 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:48.787059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:48.787390 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 04:32:48.787599 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 04:32:48.788063 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":132,"threads_started":1}
I20260806 04:32:48.789112 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:48.770009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:59949 (local address 127.28.94.62:46167)
0806 04:32:48.770455 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:48.770476 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:48.770975 (+ 499us) server_negotiation.cc:407] Connection header received
0806 04:32:48.772227 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:48.772245 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:48.772723 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:48.773219 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:48.774812 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.776323 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.778117 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.778843 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.779031 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:48.782457 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:48.782466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:48.782489 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:48.782545 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:48.784873 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:48.785637 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:48.785645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:48.785652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:48.785841 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:48.786314 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:48.786317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:48.786319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:48.786912 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:48.787078 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:48.787776 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 04:32:48.788139 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260806 04:32:48.790946 18645 raft_consensus.cc:348] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:48.791456 18645 raft_consensus.cc:729] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5930fa863a4870b7cd8f43dc80386a, State: Initialized, Role: FOLLOWER
I20260806 04:32:48.791958 18645 consensus_queue.cc:260] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } }
I20260806 04:32:48.792771 18798 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:48.793066 18798 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:48.793123 18645 ts_tablet_manager.cc:1428] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260806 04:32:48.793696 18798 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
I20260806 04:32:48.796681 18528 ts_manager.cc:194] Registered new tserver with Master: 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193)
I20260806 04:32:48.800386 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:59949
I20260806 04:32:48.803467 18798 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
Aug 06 04:32:48 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.1@KRBTEST.COM
I20260806 04:32:48.812047 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:48.799972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44193 (local address 127.0.0.1:48732)
0806 04:32:48.800957 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:48.801009 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:48.801186 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:48.801958 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:48.801965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:48.802474 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:48.802625 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:48.802635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.803982 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.803986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:48.804983 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:48.804996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.805768 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.805770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:48.805855 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:48.806428 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:48.806441 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:48.807863 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:48.810076 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:48.810086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:48.810092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:48.810396 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:48.810779 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:48.810782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:48.810784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:48.810883 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:48.811431 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:48.811436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:48.811573 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 04:32:48.811695 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 04:32:48.811891 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":120,"threads_started":1}
I20260806 04:32:48.812146 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:48.800104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.28.94.1:44193)
0806 04:32:48.800729 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:48.800742 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:48.801128 (+ 386us) server_negotiation.cc:407] Connection header received
0806 04:32:48.801535 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:48.801542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:48.801779 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:48.801926 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:48.802773 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.803844 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.805118 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:48.805645 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:48.805767 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:48.808027 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:48.808041 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:48.808057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:48.808088 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:48.809951 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:48.810525 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:48.810532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:48.810534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:48.810645 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:48.810969 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:48.810972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:48.810974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:48.811307 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:48.811442 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:48.811797 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 04:32:48.811965 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"spinlock_wait_cycles":768,"thread_start_us":128,"threads_started":1}
I20260806 04:32:48.813347 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9a8177f90f466da1e8abf2a790c28a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:48732
I20260806 04:32:48.814476 18808 tablet_replica.cc:330] stopping tablet replica
I20260806 04:32:48.814829 18808 raft_consensus.cc:2229] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:48.815093 18808 raft_consensus.cc:2258] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:48.816169 18808 ts_tablet_manager.cc:1905] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:48.820175 18808 ts_tablet_manager.cc:1918] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:48.820287 18808 log.cc:1196] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal/wals/4d9a8177f90f466da1e8abf2a790c28a
I20260806 04:32:48.820559 18808 ts_tablet_manager.cc:1939] T 4d9a8177f90f466da1e8abf2a790c28a P 2b5930fa863a4870b7cd8f43dc80386a: Deleting consensus metadata
I20260806 04:32:48.822544 18508 catalog_manager.cc:4841] TS 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193): tablet 4d9a8177f90f466da1e8abf2a790c28a (table test-table [id=024f267c36bc4ecdaeddd9ec924fc112]) successfully deleted
W20260806 04:32:48.909826 18802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:48.910163 18802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:48.910229 18802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:48.914664 18802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:48.914749 18802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:48.914837 18802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:32:48.920615 18802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:40863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=37087
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:48.921900 18802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:48.923108 18802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:48.933784 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:48.933789 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:48.934034 18802 server_base.cc:1034] running on GCE node
W20260806 04:32:48.933784 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:48.934634 18802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:48.935410 18802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:48.936581 18802 hybrid_clock.cc:648] HybridClock initialized: now 1785990768936561 us; error 40 us; skew 500 ppm
Aug 06 04:32:48 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:48.939513 18802 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:32:48.940554 18802 webserver.cc:460] Webserver started at http://127.28.94.2:37087/ using document root <none> and password file <none>
I20260806 04:32:48.941239 18802 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:48.941337 18802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:48.946010 18802 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:32:48.948500 18829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:48.949901 18802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:32:48.950057 18802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "65a98e145157184daf3665a7901532ee"
server_key_iv: "931555c2f754848be4e4c28880c8f541"
server_key_version: "encryptionkey@0"
I20260806 04:32:48.950688 18802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:48.984200 18802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:48.987534 18802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:48.987751 18802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:48.988430 18802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:48.989423 18802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:48.989480 18802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:48.989550 18802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:48.989576 18802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:48.998775 18802 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:40863
I20260806 04:32:48.998803 18996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:40863 every 8 connection(s)
I20260806 04:32:48.999887 18802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:32:49.004670 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 18802
I20260806 04:32:49.005250 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:32941
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=46313
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:32:49 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990768, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:49.015002 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.001973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:47503 (local address 127.28.94.62:46167)
0806 04:32:49.002148 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.002153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.003405 (+ 1252us) server_negotiation.cc:407] Connection header received
0806 04:32:49.004298 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.004301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.004358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.004437 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:49.006184 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.006736 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.008093 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.008343 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.008442 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.011265 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:49.011268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:49.011270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:49.011303 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:49.012733 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.013338 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.013341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.013342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.013430 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.013787 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.013790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.013791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.013996 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:49.014088 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.014661 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.014822 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:32:49.015908 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.002437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:47503)
0806 04:32:49.003212 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.003269 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.004110 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.004669 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.004680 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.005243 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:49.005966 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.005988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.006873 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.006877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.007939 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.007951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.008512 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.008516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.008717 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.009364 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:49.009384 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:49.011001 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:49.012868 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:49.012876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:49.012929 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:49.013232 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:49.013519 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:49.013522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:49.013524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:49.013683 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:49.014101 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:49.014110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:49.014400 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.014583 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.014963 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":124,"threads_started":1}
I20260806 04:32:49.017437 18997 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:49.017740 18997 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:49.018309 18997 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
I20260806 04:32:49.019575 18528 ts_manager.cc:194] Registered new tserver with Master: 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863)
I20260806 04:32:49.020376 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:47503
W20260806 04:32:49.130741 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:32:49.131023 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:32:49.131068 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:32:49.135293 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:32:49.135386 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:32:49.135447 19000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:32:49.141216 19000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:32941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=46313
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:32:49.142463 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:32:49.143683 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:32:49.154371 19008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:32:49.154367 19007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:49.154614 19000 server_base.cc:1034] running on GCE node
W20260806 04:32:49.154367 19010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:32:49.155305 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:32:49.156076 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:32:49.157279 19000 hybrid_clock.cc:648] HybridClock initialized: now 1785990769157252 us; error 37 us; skew 500 ppm
Aug 06 04:32:49 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990769, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:32:49.160321 19000 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:32:49.161394 19000 webserver.cc:460] Webserver started at http://127.28.94.3:46313/ using document root <none> and password file <none>
I20260806 04:32:49.161985 19000 fs_manager.cc:359] Metadata directory not provided
I20260806 04:32:49.162073 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:32:49.166957 19000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:32:49.169346 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:49.170799 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:32:49.170926 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "fdf37a8e765342a5b5a1f07712f9218b"
format_stamp: "Formatted at 2026-08-06 04:32:28 on dist-test-slave-ncp9"
server_key: "97e38ce4db078a446d4c21efdfda153c"
server_key_iv: "a218385951921d5964b519edff906ed5"
server_key_version: "encryptionkey@0"
I20260806 04:32:49.171517 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:32:49.184768 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:32:49.188169 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:32:49.188349 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:32:49.189023 19000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:32:49.189942 19000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:32:49.189993 19000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:49.190037 19000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:32:49.190057 19000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:32:49.199317 19000 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:32941
I20260806 04:32:49.199344 19183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:32941 every 8 connection(s)
I20260806 04:32:49.200423 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:32:49.203864 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 19000
Aug 06 04:32:49 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990769, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:32:49.212749 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.204549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.28.94.62:46167)
0806 04:32:49.204785 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.204790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.205003 (+ 213us) server_negotiation.cc:407] Connection header received
0806 04:32:49.205082 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.205085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.205133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.205206 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:49.206010 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.206549 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.207411 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.207645 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.207735 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.209421 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:49.209424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:49.209426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:49.209450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:49.211012 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.211460 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.211463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.211464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.211546 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.211826 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.211829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.211830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.212022 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:49.212187 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.212410 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.212547 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
I20260806 04:32:49.214836 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.202484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:33563 (local address 127.28.94.62:46167)
0806 04:32:49.202690 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.202694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.203842 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 04:32:49.204862 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.204865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.204940 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.205030 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:49.206323 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.206821 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.207980 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.208256 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.208358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.210912 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:49.210915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:49.210917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:49.210954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:49.212295 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.212934 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.212938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.212940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.213028 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.213573 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.213577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.213578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.213790 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:49.213883 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.214536 (+ 653us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.214705 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 04:32:49.215862 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.202917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:33563)
0806 04:32:49.203659 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.203704 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.204621 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.205254 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.205261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.205632 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:32:49.206161 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.206176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.206946 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.206950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.207859 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.207869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.208416 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.208422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.208643 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.209185 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:32:49.209208 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:32:49.210737 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:32:49.212410 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:49.212418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:49.212438 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:49.212790 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:49.213279 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:32:49.213282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:32:49.213284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:32:49.213469 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:32:49.213894 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:32:49.213902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:32:49.214259 (+ 357us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.214488 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.214887 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":135,"threads_started":1}
I20260806 04:32:49.217087 19184 heartbeater.cc:347] Connected to a master server at 127.28.94.62:46167
I20260806 04:32:49.217331 19184 heartbeater.cc:464] Registering TS with master...
I20260806 04:32:49.217888 19184 heartbeater.cc:511] Master 127.28.94.62:46167 requested a full tablet report, sending...
I20260806 04:32:49.218976 18528 ts_manager.cc:194] Registered new tserver with Master: fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
I20260806 04:32:49.219558 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:33563
I20260806 04:32:49.226866 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:32:49.231385 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.227548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.28.94.62:46167)
0806 04:32:49.227696 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.227701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.227909 (+ 208us) server_negotiation.cc:407] Connection header received
0806 04:32:49.228092 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.228095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.228212 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.228310 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:49.228597 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.229240 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.230151 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.230382 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.230472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.230801 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:49.230960 (+ 159us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 04:32:49.231238 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45316: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 04:32:49.231477 18800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45316: token expired
W20260806 04:32:49.231813 18425 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.94.62:46167)
I20260806 04:32:49.238889 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.232188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.28.94.62:46167)
0806 04:32:49.232305 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.232309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.232322 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:32:49.232400 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.232403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.232447 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.232509 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:32:49.233156 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.233647 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.234426 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.234650 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.234734 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.236038 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:32:49.236042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:32:49.236045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:32:49.236072 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:32:49.237299 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.237701 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.237705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.237707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.237790 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:32:49.238038 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:32:49.238041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:32:49.238043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:32:49.238235 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:32:49.238361 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.238605 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.238729 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:32:49.241644 18528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:32:49.245350 18528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:32:49.254102 18705 tablet_service.cc:1467] Processing CreateTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 (DEFAULT_TABLE table=test-table [id=a90e3a126d854179ae500a1cb2a0cbad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:49.254944 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b92e0af67b4e02b737d690a052b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:49.258088 18645 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap starting.
I20260806 04:32:49.258932 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.253017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:32941 (local address 127.0.0.1:33974)
0806 04:32:49.253314 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.253327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.253408 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.254279 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.254282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.254300 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:49.254383 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.254388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.255844 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.255848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.257064 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.257077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.258456 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.258460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.258550 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.258616 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.258685 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.258772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":246,"thread_start_us":126,"threads_started":1}
I20260806 04:32:49.259083 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.252759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:40863 (local address 127.0.0.1:47838)
0806 04:32:49.253017 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.253032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.253120 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.254072 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.254075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.254095 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:49.254218 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.254225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.255825 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.255830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.257249 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.257262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.258647 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.258651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.258752 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.258781 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.258865 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.258953 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":214}
I20260806 04:32:49.259088 18645 tablet_bootstrap.cc:654] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:49.259521 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.252995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.28.94.2:40863)
0806 04:32:49.253668 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.253678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.253699 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:32:49.253763 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.253767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.253928 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.254076 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:49.254429 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.255687 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.257394 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.258498 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.258612 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.258758 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.258911 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.259375 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":148,"threads_started":1}
I20260806 04:32:49.260042 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.253246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.28.94.3:32941)
0806 04:32:49.253568 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.253588 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.253615 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:32:49.253706 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.253712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.253946 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.254104 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:49.254506 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.255678 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.257205 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.258084 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.258204 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.258386 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.259377 (+ 991us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.259900 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
I20260806 04:32:49.260983 18645 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: No bootstrap required, opened a new log
I20260806 04:32:49.261090 18645 ts_tablet_manager.cc:1397] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:32:49.261639 18645 raft_consensus.cc:348] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.261777 18645 raft_consensus.cc:374] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:49.261816 18645 raft_consensus.cc:729] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5930fa863a4870b7cd8f43dc80386a, State: Initialized, Role: FOLLOWER
I20260806 04:32:49.261992 18645 consensus_queue.cc:260] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.262439 18645 ts_tablet_manager.cc:1428] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:32:49.263299 18898 tablet_service.cc:1467] Processing CreateTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 (DEFAULT_TABLE table=test-table [id=a90e3a126d854179ae500a1cb2a0cbad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:49.263306 19094 tablet_service.cc:1467] Processing CreateTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 (DEFAULT_TABLE table=test-table [id=a90e3a126d854179ae500a1cb2a0cbad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:32:49.264472 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b92e0af67b4e02b737d690a052b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:32:49.264467 18898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b92e0af67b4e02b737d690a052b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 04:32:49.269071 18799 tablet.cc:2369] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:49.272202 19206 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Bootstrap starting.
I20260806 04:32:49.272202 19205 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Bootstrap starting.
I20260806 04:32:49.274963 19206 tablet_bootstrap.cc:654] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:49.274973 19205 tablet_bootstrap.cc:654] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Neither blocks nor log segments found. Creating new log.
I20260806 04:32:49.275866 19206 log.cc:824] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:49.275866 19205 log.cc:824] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:32:49.278012 19205 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: No bootstrap required, opened a new log
I20260806 04:32:49.278016 19206 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: No bootstrap required, opened a new log
I20260806 04:32:49.278228 19206 ts_tablet_manager.cc:1397] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:32:49.278229 19205 ts_tablet_manager.cc:1397] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:32:49.281550 19205 raft_consensus.cc:348] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.281503 19206 raft_consensus.cc:348] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.281831 19206 raft_consensus.cc:374] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:49.281832 19205 raft_consensus.cc:374] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:32:49.281898 19206 raft_consensus.cc:729] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf37a8e765342a5b5a1f07712f9218b, State: Initialized, Role: FOLLOWER
I20260806 04:32:49.281911 19205 raft_consensus.cc:729] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7c5f3ee3f4493a24fad90d9a991c4, State: Initialized, Role: FOLLOWER
I20260806 04:32:49.282398 19206 consensus_queue.cc:260] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.282397 19205 consensus_queue.cc:260] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.283253 19184 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
I20260806 04:32:49.283423 19206 ts_tablet_manager.cc:1428] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:32:49.283485 19205 ts_tablet_manager.cc:1428] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 04:32:49.283819 18997 heartbeater.cc:503] Master 127.28.94.62:46167 was elected leader, sending a full tablet report...
W20260806 04:32:49.451494 19187 tablet.cc:2369] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:32:49.501133 18998 tablet.cc:2369] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:32:49.543035 19210 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:32:49.543352 19210 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.544662 19210 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941), 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193)
I20260806 04:32:49.548648 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.545068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:38373 (local address 127.28.94.3:32941)
0806 04:32:49.545234 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.545238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.545380 (+ 142us) server_negotiation.cc:407] Connection header received
0806 04:32:49.545589 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.545593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.545653 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.545695 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:49.546064 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.546566 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.547699 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.548099 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.548198 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.548267 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.548440 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.548514 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:32:49.548663 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.544971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:32941 (local address 127.28.94.2:38373)
0806 04:32:49.545289 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.545304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.545427 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.545828 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.545830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.545849 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:49.545952 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.545956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.546688 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.546691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.547569 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.547579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.548226 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.548229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.548326 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.548347 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.548412 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.548489 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":35,"thread_start_us":95,"threads_started":1}
I20260806 04:32:49.548979 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.545399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44193 (local address 127.28.94.2:36603)
0806 04:32:49.545664 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.545680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:32:49.545804 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:32:49.546120 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:32:49.546124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:32:49.546141 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:32:49.546265 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.546271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.547009 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.547013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.547828 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:32:49.547838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.548530 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.548533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:32:49.548631 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.548654 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:32:49.548711 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 04:32:49.548791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":19,"thread_start_us":88,"threads_started":1}
I20260806 04:32:49.549381 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.545549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:36603 (local address 127.28.94.1:44193)
0806 04:32:49.545862 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.545866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.545880 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:32:49.545933 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.545936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.545988 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.546085 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:32:49.546382 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.546892 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.547961 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.548404 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.548501 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.548685 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.549192 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.549275 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260806 04:32:49.549417 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b92e0af67b4e02b737d690a052b4f9" candidate_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf37a8e765342a5b5a1f07712f9218b" is_pre_election: true
I20260806 04:32:49.549793 19122 raft_consensus.cc:2454] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb7c5f3ee3f4493a24fad90d9a991c4 in term 0.
I20260806 04:32:49.550026 18737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b92e0af67b4e02b737d690a052b4f9" candidate_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5930fa863a4870b7cd8f43dc80386a" is_pre_election: true
I20260806 04:32:49.550357 18737 raft_consensus.cc:2454] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb7c5f3ee3f4493a24fad90d9a991c4 in term 0.
I20260806 04:32:49.550352 18832 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b; no voters:
I20260806 04:32:49.550642 19210 raft_consensus.cc:2781] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:32:49.550742 19210 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:32:49.550794 19210 raft_consensus.cc:3037] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:49.551940 19210 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.552294 19210 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [CANDIDATE]: Term 1 election: Requested vote from peers fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941), 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193)
I20260806 04:32:49.552755 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b92e0af67b4e02b737d690a052b4f9" candidate_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf37a8e765342a5b5a1f07712f9218b"
I20260806 04:32:49.552755 18737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b92e0af67b4e02b737d690a052b4f9" candidate_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5930fa863a4870b7cd8f43dc80386a"
I20260806 04:32:49.552919 18737 raft_consensus.cc:3037] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:49.552920 19122 raft_consensus.cc:3037] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:32:49.554255 19122 raft_consensus.cc:2454] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb7c5f3ee3f4493a24fad90d9a991c4 in term 1.
I20260806 04:32:49.554255 18737 raft_consensus.cc:2454] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb7c5f3ee3f4493a24fad90d9a991c4 in term 1.
I20260806 04:32:49.554569 18832 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b; no voters:
I20260806 04:32:49.554757 19210 raft_consensus.cc:2781] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:32:49.554975 19210 raft_consensus.cc:686] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb7c5f3ee3f4493a24fad90d9a991c4, State: Running, Role: LEADER
I20260806 04:32:49.555361 19210 consensus_queue.cc:237] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:32:49.558841 18528 catalog_manager.cc:5495] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2). New cstate: current_term: 1 leader_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } health_report { overall_health: UNKNOWN } } }
I20260806 04:32:49.630378 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.627004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.28.94.2:40863)
0806 04:32:49.627137 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:32:49.627141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:32:49.627156 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:32:49.627282 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:32:49.627285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:32:49.627347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:32:49.627442 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:32:49.627632 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.628188 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.629011 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:32:49.629225 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:32:49.629302 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:32:49.629553 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:32:49.630036 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:32:49.630143 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:32:49.630189 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 04:32:49.630252 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:32:49.637241 19122 raft_consensus.cc:1261] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Refusing update from remote peer 2cb7c5f3ee3f4493a24fad90d9a991c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:49.637238 18737 raft_consensus.cc:1261] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Refusing update from remote peer 2cb7c5f3ee3f4493a24fad90d9a991c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:32:49.638088 19214 consensus_queue.cc:1035] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:49.638309 19210 consensus_queue.cc:1035] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:32:49.646085 19220 mvcc.cc:205] Tried to move back new op lower bound from 7315418192424960000 to 7315418192101785600. Current Snapshot: MvccSnapshot[applied={T|T < 7315418192424960000}]
I20260806 04:32:49.657162 18524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:49.657400 18524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:32:49.660266 18524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Sending DeleteTablet for 3 replicas of tablet 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:32:49.661259 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:49 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:48732
I20260806 04:32:49.661314 19094 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:49 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:33974
I20260806 04:32:49.661532 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 18585
I20260806 04:32:49.661818 18898 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b92e0af67b4e02b737d690a052b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:32:49 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:47838
I20260806 04:32:49.665198 19233 tablet_replica.cc:330] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: stopping tablet replica
I20260806 04:32:49.665649 19233 raft_consensus.cc:2229] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:32:49.666162 19233 raft_consensus.cc:2258] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:49.666631 19232 tablet_replica.cc:330] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: stopping tablet replica
I20260806 04:32:49.666978 19232 raft_consensus.cc:2229] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:32:49.667389 19232 raft_consensus.cc:2258] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:32:49.668154 19233 ts_tablet_manager.cc:1905] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:49.669137 19232 ts_tablet_manager.cc:1905] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:32:49.671198 19233 ts_tablet_manager.cc:1918] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:49.671301 19233 log.cc:1196] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/71b92e0af67b4e02b737d690a052b4f9
I20260806 04:32:49.671602 19233 ts_tablet_manager.cc:1939] T 71b92e0af67b4e02b737d690a052b4f9 P 2cb7c5f3ee3f4493a24fad90d9a991c4: Deleting consensus metadata
I20260806 04:32:49.671857 19232 ts_tablet_manager.cc:1918] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:32:49.671949 19232 log.cc:1196] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/71b92e0af67b4e02b737d690a052b4f9
I20260806 04:32:49.672260 19232 ts_tablet_manager.cc:1939] T 71b92e0af67b4e02b737d690a052b4f9 P fdf37a8e765342a5b5a1f07712f9218b: Deleting consensus metadata
I20260806 04:32:49.673277 18508 catalog_manager.cc:4841] TS 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): tablet 71b92e0af67b4e02b737d690a052b4f9 (table test-table [id=a90e3a126d854179ae500a1cb2a0cbad]) successfully deleted
I20260806 04:32:49.673605 18508 catalog_manager.cc:4841] TS fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): tablet 71b92e0af67b4e02b737d690a052b4f9 (table test-table [id=a90e3a126d854179ae500a1cb2a0cbad]) successfully deleted
W20260806 04:32:49.675822 18508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:44193 (error 108)
I20260806 04:32:49.676056 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 18802
I20260806 04:32:49.680758 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:32:49.680342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44193 (local address 127.0.0.1:48740)
0806 04:32:49.680569 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 04:32:49.680663 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 04:32:49.680979 18508 catalog_manager.cc:4584] TS 2b5930fa863a4870b7cd8f43dc80386a (127.28.94.1:44193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71b92e0af67b4e02b737d690a052b4f9: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44193: connect: Connection refused (error 111)
I20260806 04:32:49.687861 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 19000
I20260806 04:32:49.701459 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 18477
I20260806 04:33:09.720187 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--webserver_interface=127.28.94.62
--webserver_port=40013
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:09.841881 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:09.842238 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:09.842299 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:09.846654 19236 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:33:09.846742 19236 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:33:09.846764 19236 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:33:09.846781 19236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:09.846804 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:09.846830 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:09.846860 19236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:09.846884 19236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:09.852710 19236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:09.854003 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:09.855134 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:09.864511 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:09.864605 19236 server_base.cc:1034] running on GCE node
W20260806 04:33:09.864521 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:09.864511 19248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:09.865509 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:09.866776 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:09.867940 19236 hybrid_clock.cc:648] HybridClock initialized: now 1785990789867920 us; error 33 us; skew 500 ppm
Aug 06 04:33:09 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990789, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:09.874212 19236 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:09.875353 19236 webserver.cc:460] Webserver started at http://127.28.94.62:40013/ using document root <none> and password file <none>
I20260806 04:33:09.876009 19236 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:09.876094 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:09.880722 19236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:09.883277 19264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:09.884640 19236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:09.884788 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "19a05c279a5b444e8f06e75dba5380d7"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "2f74b4d005a3b989c0681523a58b2e00"
server_key_iv: "fc5b908121314d917894e8f0463d9a4c"
server_key_version: "encryptionkey@0"
I20260806 04:33:09.885454 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:09.903568 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:09.906677 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:09.906890 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:09.914610 19236 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:46167
I20260806 04:33:09.914945 19332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:46167 every 8 connection(s)
I20260806 04:33:09.916167 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:09.918869 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 19236
I20260806 04:33:09.919404 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:44193
--local_ip_for_outbound_sockets=127.28.94.1
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=37443
--webserver_interface=127.28.94.1
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:33:09.921996 19333 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:33:09.924008 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap starting.
I20260806 04:33:09.943419 19333 log.cc:824] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:09.955938 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:33:09.956377 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Bootstrap complete.
I20260806 04:33:09.960685 19333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:33:09.961213 19333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Initialized, Role: FOLLOWER
I20260806 04:33:09.961774 19333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:33:09.961907 19333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:09.961972 19333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:09.962081 19333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:33:09.964308 19333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:33:09.964679 19333 leader_election.cc:302] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a05c279a5b444e8f06e75dba5380d7; no voters:
I20260806 04:33:09.965072 19333 leader_election.cc:288] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:33:09.965204 19338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:33:09.965417 19338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [term 3 LEADER]: Becoming Leader. State: Replica: 19a05c279a5b444e8f06e75dba5380d7, State: Running, Role: LEADER
I20260806 04:33:09.965765 19338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } }
I20260806 04:33:09.966537 19333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:09.966918 19338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a05c279a5b444e8f06e75dba5380d7. Latest consensus state: current_term: 3 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:33:09.967192 19338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:09.968379 19348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:09.968694 19340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "19a05c279a5b444e8f06e75dba5380d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a05c279a5b444e8f06e75dba5380d7" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 46167 } } }
I20260806 04:33:09.968802 19340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:09.974148 19348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:09.975198 19348 catalog_manager.cc:1259] Loaded cluster ID: b2750799c07542da9cd96b56244a1bb2
I20260806 04:33:09.975282 19348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:09.978094 19348 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:09.979033 19348 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Loaded TSK: 0
I20260806 04:33:09.979140 19348 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19a05c279a5b444e8f06e75dba5380d7: Loaded TSK: 1
F20260806 04:33:09.980038 19348 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbb98ae39ad google::LogMessage::Fail() at ??:0
@ 0x7fbb98ae688a google::LogMessage::SendToLog() at ??:0
@ 0x7fbb98ae34de google::LogMessage::Flush() at ??:0
@ 0x7fbb98ae4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb9b7a718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbb9b79ff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbba34487bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbba342299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbba3423178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbba3464df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbba348382b std::function<>::operator()() at ??:0
@ 0x7fbba3422b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbba3423f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbba3420579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbba3464640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbb9ad353d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb9ad35c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbb9ad373f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbb9ad26512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbb9c36a6db start_thread at ??:0
@ 0x7fbb975e161f clone at ??:0
W20260806 04:33:10.064857 19336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:10.065173 19336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:10.065220 19336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:10.069764 19336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:10.069824 19336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:10.069887 19336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:10.075892 19336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:44193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=37443
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:10.077329 19336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:10.078421 19336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:10.089282 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:10.089416 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:10.089293 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:10.089574 19336 server_base.cc:1034] running on GCE node
I20260806 04:33:10.090174 19336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:10.090898 19336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:10.092089 19336 hybrid_clock.cc:648] HybridClock initialized: now 1785990790092079 us; error 36 us; skew 500 ppm
Aug 06 04:33:10 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990790, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:10.095299 19336 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:33:10.096421 19336 webserver.cc:460] Webserver started at http://127.28.94.1:37443/ using document root <none> and password file <none>
I20260806 04:33:10.097131 19336 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:10.097214 19336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:10.101660 19336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:10.104074 19419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.105537 19336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:10.105691 19336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "2b5930fa863a4870b7cd8f43dc80386a"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "1fafe7fa2c9237be1b92972a8ee6f411"
server_key_iv: "d8e2dadce4c1c085764e589d3f0afacd"
server_key_version: "encryptionkey@0"
I20260806 04:33:10.106295 19336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:10.117702 19336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:10.121245 19336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:10.121449 19336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:10.122116 19336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:10.123399 19427 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:33:10.129619 19336 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:33:10.129715 19336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 04:33:10.129791 19336 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:33:10.131758 19336 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:33:10.131821 19336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:33:10.132215 19427 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap starting.
I20260806 04:33:10.142230 19336 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44193
I20260806 04:33:10.142253 19589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44193 every 8 connection(s)
I20260806 04:33:10.143383 19336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:10.148330 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 19336
I20260806 04:33:10.148790 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:40863
--local_ip_for_outbound_sockets=127.28.94.2
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=37087
--webserver_interface=127.28.94.2
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:33:10.155694 19427 log.cc:824] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:10.166124 19427 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:33:10.166653 19427 tablet_bootstrap.cc:492] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Bootstrap complete.
I20260806 04:33:10.168129 19427 ts_tablet_manager.cc:1397] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.012s
I20260806 04:33:10.171870 19427 raft_consensus.cc:348] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:10.172494 19427 raft_consensus.cc:729] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5930fa863a4870b7cd8f43dc80386a, State: Initialized, Role: FOLLOWER
I20260806 04:33:10.173120 19427 consensus_queue.cc:260] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:10.174320 19427 ts_tablet_manager.cc:1428] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260806 04:33:10.238296 19593 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:33:10.145670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:45667)
0806 04:33:10.146227 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.146287 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:10.147140 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:10.237169 (+ 90029us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":135,"threads_started":1}
W20260806 04:33:10.238945 19590 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46167 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46167: Client connection negotiation failed: client connection to 127.28.94.62:46167: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:33:10.239806 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.239481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:40207)
0806 04:33:10.239633 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.239722 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 04:33:10.240526 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.240291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:51979)
0806 04:33:10.240405 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.240458 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:33:10.240710 19590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:33:10.277232 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:10.277554 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:10.277599 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:10.281812 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:10.281894 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:10.281957 19594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:10.287969 19594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:40863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=37087
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:10.289211 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:10.290351 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:10.300772 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:10.300755 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:10.301010 19594 server_base.cc:1034] running on GCE node
W20260806 04:33:10.300791 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:10.301699 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:10.302451 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:10.303635 19594 hybrid_clock.cc:648] HybridClock initialized: now 1785990790303611 us; error 33 us; skew 500 ppm
Aug 06 04:33:10 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990790, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:10.306643 19594 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:33:10.307698 19594 webserver.cc:460] Webserver started at http://127.28.94.2:37087/ using document root <none> and password file <none>
I20260806 04:33:10.308324 19594 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:10.308403 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:10.312923 19594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:10.315349 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.316809 19594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:10.316983 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4"
format_stamp: "Formatted at 2026-08-06 04:32:27 on dist-test-slave-ncp9"
server_key: "65a98e145157184daf3665a7901532ee"
server_key_iv: "931555c2f754848be4e4c28880c8f541"
server_key_version: "encryptionkey@0"
I20260806 04:33:10.317595 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:10.331509 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:10.334702 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:10.334911 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:10.335577 19594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:10.336509 19594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:10.336571 19594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.336632 19594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:10.336658 19594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.346300 19594 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:40863
I20260806 04:33:10.346324 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:40863 every 8 connection(s)
I20260806 04:33:10.347494 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:10.351704 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.349881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:39751)
0806 04:33:10.350639 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.350765 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":133,"threads_started":1}
W20260806 04:33:10.352248 19773 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46167 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46167: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
I20260806 04:33:10.352322 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 19594
I20260806 04:33:10.352852 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:32941
--local_ip_for_outbound_sockets=127.28.94.3
--tserver_master_addrs=127.28.94.62:46167
--webserver_port=46313
--webserver_interface=127.28.94.3
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:33:10.353053 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.352744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:55973)
0806 04:33:10.352869 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.352961 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:33:10.353670 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.353477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:35805)
0806 04:33:10.353577 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.353619 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 04:33:10.353826 19773 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:33:10.475804 19778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:10.476125 19778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:10.476174 19778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:10.480420 19778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:10.480507 19778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:10.480588 19778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:10.486370 19778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:32941
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=46313
--tserver_master_addrs=127.28.94.62:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:10.487617 19778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:10.488731 19778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:10.499183 19790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:10.499239 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:10.499248 19785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:10.499539 19778 server_base.cc:1034] running on GCE node
I20260806 04:33:10.499981 19778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:10.500748 19778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:10.501895 19778 hybrid_clock.cc:648] HybridClock initialized: now 1785990790501860 us; error 54 us; skew 500 ppm
Aug 06 04:33:10 dist-test-slave-ncp9 krb5kdc[17690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990790, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:10.504760 19778 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:33:10.505800 19778 webserver.cc:460] Webserver started at http://127.28.94.3:46313/ using document root <none> and password file <none>
I20260806 04:33:10.506556 19778 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:10.506661 19778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:10.511255 19778 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:10.513639 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.515044 19778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:10.515179 19778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "fdf37a8e765342a5b5a1f07712f9218b"
format_stamp: "Formatted at 2026-08-06 04:32:28 on dist-test-slave-ncp9"
server_key: "97e38ce4db078a446d4c21efdfda153c"
server_key_iv: "a218385951921d5964b519edff906ed5"
server_key_version: "encryptionkey@0"
I20260806 04:33:10.515770 19778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:10.530956 19778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:10.534154 19778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:10.534351 19778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:10.535007 19778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:10.535938 19778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:10.535986 19778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.536031 19778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:10.536059 19778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:10.545300 19778 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:32941
I20260806 04:33:10.545353 19971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:32941 every 8 connection(s)
I20260806 04:33:10.546458 19778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:10.550853 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.548792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:34265)
0806 04:33:10.549363 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.549482 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":136,"threads_started":1}
I20260806 04:33:10.551422 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 19778
W20260806 04:33:10.551563 19972 heartbeater.cc:644] Failed to heartbeat to 127.28.94.62:46167 (0 consecutive failures): Network error: Failed to ping master at 127.28.94.62:46167: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
I20260806 04:33:10.552302 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.552048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:58763)
0806 04:33:10.552180 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.552236 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:33:10.552974 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:10.552756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:54837)
0806 04:33:10.552850 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:10.552912 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 04:33:10.553131 19972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:33:11.241958 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.241473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:52801)
0806 04:33:11.241777 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:11.241832 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260806 04:33:11.354866 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.354418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:47427)
0806 04:33:11.354676 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:11.354736 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260806 04:33:11.462574 19980 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:11.462848 19980 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:11.464169 19980 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863), fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
Aug 06 04:33:11 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990790, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.2@KRBTEST.COM
Aug 06 04:33:11 dist-test-slave-ncp9 krb5kdc[17690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990790, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.3@KRBTEST.COM
I20260806 04:33:11.479672 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.464426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:40863 (local address 127.28.94.1:38175)
0806 04:33:11.464575 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:11.464591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:11.464727 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:11.466687 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:11.466694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:11.467252 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:11.467787 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:11.467803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.469146 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.469149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:11.470176 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:11.470187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.470913 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:11.471164 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:11.471855 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:11.471868 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:11.473532 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:11.476370 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:11.476379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:11.476396 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:11.476949 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:11.477357 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:11.477362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:11.477593 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:11.478413 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:11.478422 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:11.478797 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 04:33:11.479026 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:33:11.479516 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":313,"spinlock_wait_cycles":126848}
I20260806 04:33:11.479708 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.464580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:38175 (local address 127.28.94.2:40863)
0806 04:33:11.465115 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:11.465152 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:11.465202 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:33:11.465910 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:11.465916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:11.466293 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:11.466585 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:11.467979 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.468998 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470301 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.470783 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470949 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:11.473801 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:11.473807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:11.473822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:11.473858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:11.476216 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477069 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:11.477076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:11.477082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:11.477260 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477689 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:11.477692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:11.477693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:11.478294 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:11.478437 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:11.479172 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 04:33:11.479500 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":147,"threads_started":1}
I20260806 04:33:11.479707 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.464959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:48891 (local address 127.28.94.3:32941)
0806 04:33:11.465307 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:11.465317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:11.465349 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:33:11.465979 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:11.465985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:11.466299 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:11.466588 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:11.467968 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.469021 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470299 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.470781 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470949 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:11.474068 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:11.474075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:11.474091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:11.474129 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:11.476216 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477069 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:11.477076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:11.477082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:11.477260 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477701 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:11.477704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:11.477705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:11.478294 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:11.478437 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:11.479170 (+ 733us) server_negotiation.cc:299] Negotiation successful
0806 04:33:11.479501 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260806 04:33:11.479671 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.464733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:32941 (local address 127.28.94.1:48891)
0806 04:33:11.465053 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:11.465065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:11.465158 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:11.466687 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:11.466694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:11.467251 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:11.467786 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:11.467802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.469157 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.469159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:11.470177 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:11.470188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:11.470913 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:11.470917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:11.471168 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:11.471807 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:11.471826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:11.473841 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:11.476370 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:11.476379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:11.476396 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:11.476949 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:11.477357 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:11.477360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:11.477362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:11.477593 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:11.478413 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:11.478422 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:11.478797 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 04:33:11.479026 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:33:11.479444 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":215,"spinlock_wait_cycles":48512,"thread_start_us":79,"threads_started":1}
W20260806 04:33:11.482364 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
W20260806 04:33:11.482602 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:33:11.482700 19422 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a; no voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b
I20260806 04:33:11.482988 19980 raft_consensus.cc:2726] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:33:11.554172 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:11.553682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:46509)
0806 04:33:11.553995 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:11.554059 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
I20260806 04:33:12.243280 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:12.242758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:58119)
0806 04:33:12.243079 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:12.243170 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260806 04:33:12.356307 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:12.355814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:57797)
0806 04:33:12.356134 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:12.356204 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260806 04:33:12.555596 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:12.555068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:45971)
0806 04:33:12.555395 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:12.555469 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
I20260806 04:33:13.244657 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:13.244180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:55339)
0806 04:33:13.244473 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:13.244544 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260806 04:33:13.357700 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:13.357190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:33227)
0806 04:33:13.357496 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:13.357571 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260806 04:33:13.557147 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:13.556556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:41969)
0806 04:33:13.556966 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:13.557037 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260806 04:33:13.641363 19994 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:13.641513 19994 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:13.641925 19994 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863), fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
W20260806 04:33:13.642868 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
W20260806 04:33:13.643016 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:33:13.643071 19422 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a; no voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b
I20260806 04:33:13.643280 19994 raft_consensus.cc:2726] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:33:14.246076 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:14.245573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:33367)
0806 04:33:14.245902 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:14.245970 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260806 04:33:14.359022 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:14.358544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:40363)
0806 04:33:14.358851 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:14.358915 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260806 04:33:14.558527 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:14.558022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:48491)
0806 04:33:14.558351 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:14.558420 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260806 04:33:15.247758 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:15.247189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:53119)
0806 04:33:15.247565 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:15.247651 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
I20260806 04:33:15.360500 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:15.359963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:40749)
0806 04:33:15.360316 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:15.360390 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260806 04:33:15.560016 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:15.559486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:60797)
0806 04:33:15.559837 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:15.559907 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260806 04:33:16.018863 20002 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:16.019047 20002 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:16.019568 20002 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863), fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
W20260806 04:33:16.020454 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
W20260806 04:33:16.020587 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:33:16.020642 19422 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a; no voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b
I20260806 04:33:16.020816 20002 raft_consensus.cc:2726] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:33:16.249257 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:16.248717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:45613)
0806 04:33:16.249075 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:16.249145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":90,"threads_started":1}
I20260806 04:33:16.361991 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:16.361506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:51861)
0806 04:33:16.361805 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:16.361878 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20260806 04:33:16.561559 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:16.561071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:55779)
0806 04:33:16.561367 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:16.561441 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":104,"threads_started":1}
I20260806 04:33:17.250640 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:17.250118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:40167)
0806 04:33:17.250458 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:17.250522 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
I20260806 04:33:17.363471 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:17.362953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:54751)
0806 04:33:17.363295 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:17.363363 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":81,"threads_started":1}
I20260806 04:33:17.563117 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:17.562558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:41881)
0806 04:33:17.562934 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:17.563007 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":97,"threads_started":1}
I20260806 04:33:18.252074 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:18.251545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:46143)
0806 04:33:18.251882 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:18.251954 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
I20260806 04:33:18.365018 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:18.364465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:54531)
0806 04:33:18.364790 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:18.364862 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260806 04:33:18.380333 20015 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:18.380491 20015 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:18.380966 20015 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863), fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
W20260806 04:33:18.381871 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
W20260806 04:33:18.381991 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:33:18.382042 19422 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a; no voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b
I20260806 04:33:18.382251 20015 raft_consensus.cc:2726] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:33:18.564613 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:18.564081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:47343)
0806 04:33:18.564433 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:18.564506 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
I20260806 04:33:19.253623 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:19.253135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:60431)
0806 04:33:19.253456 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:19.253512 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260806 04:33:19.366356 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:19.365875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:37527)
0806 04:33:19.366186 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:19.366251 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260806 04:33:19.565812 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:19.565359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:34037)
0806 04:33:19.565643 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:19.565703 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
I20260806 04:33:20.255088 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:20.254628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:46579)
0806 04:33:20.254908 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:20.254975 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260806 04:33:20.367767 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:20.367309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:48213)
0806 04:33:20.367582 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:20.367661 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260806 04:33:20.567224 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:20.566727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:37785)
0806 04:33:20.567034 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:20.567114 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260806 04:33:21.256498 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:21.255965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:53663)
0806 04:33:21.256303 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:21.256378 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
I20260806 04:33:21.369292 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:21.368716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:49131)
0806 04:33:21.369095 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:21.369182 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":87,"threads_started":1}
I20260806 04:33:21.568572 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:21.568103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:59575)
0806 04:33:21.568403 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:21.568466 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260806 04:33:22.257956 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:22.257480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:52217)
0806 04:33:22.257781 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:22.257849 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260806 04:33:22.370826 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:22.370294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:50813)
0806 04:33:22.370626 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:22.370702 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260806 04:33:22.569834 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:22.569371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:56327)
0806 04:33:22.569675 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:22.569731 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260806 04:33:23.259327 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:23.258828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:38001)
0806 04:33:23.259158 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:23.259219 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260806 04:33:23.372239 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:23.371685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:48483)
0806 04:33:23.372046 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:23.372128 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260806 04:33:23.571211 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:23.570753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:57583)
0806 04:33:23.571036 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:23.571105 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260806 04:33:24.074710 20039 raft_consensus.cc:482] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:24.074872 20039 raft_consensus.cc:504] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb7c5f3ee3f4493a24fad90d9a991c4" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 40863 } } peers { permanent_uuid: "fdf37a8e765342a5b5a1f07712f9218b" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 32941 } } peers { permanent_uuid: "2b5930fa863a4870b7cd8f43dc80386a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44193 } }
I20260806 04:33:24.075318 20039 leader_election.cc:288] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863), fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941)
W20260806 04:33:24.076195 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fdf37a8e765342a5b5a1f07712f9218b (127.28.94.3:32941): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
W20260806 04:33:24.076318 19422 leader_election.cc:341] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2cb7c5f3ee3f4493a24fad90d9a991c4 (127.28.94.2:40863): Not found: Tablet not found: 71b92e0af67b4e02b737d690a052b4f9
I20260806 04:33:24.076354 19422 leader_election.cc:302] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b5930fa863a4870b7cd8f43dc80386a; no voters: 2cb7c5f3ee3f4493a24fad90d9a991c4, fdf37a8e765342a5b5a1f07712f9218b
I20260806 04:33:24.076507 20039 raft_consensus.cc:2726] T 71b92e0af67b4e02b737d690a052b4f9 P 2b5930fa863a4870b7cd8f43dc80386a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:33:24.260699 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:24.260219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:47597)
0806 04:33:24.260522 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:24.260592 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260806 04:33:24.373793 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:24.373298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:40789)
0806 04:33:24.373618 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:24.373688 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":108,"threads_started":1}
I20260806 04:33:24.572676 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:24.572127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:34259)
0806 04:33:24.572471 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:24.572543 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260806 04:33:25.262487 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:25.261950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.1:60773)
0806 04:33:25.262280 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:25.262353 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260806 04:33:25.375257 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:25.374781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.2:50453)
0806 04:33:25.375078 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:25.375151 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260806 04:33:25.561059 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 19336
I20260806 04:33:25.574298 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:25.573807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:46167 (local address 127.28.94.3:60717)
0806 04:33:25.574117 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:25.574187 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.62:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20260806 04:33:25.575371 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 19594
I20260806 04:33:25.593887 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 19778
2026-08-06T04:33:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59593 ms)
[----------] 30 tests from SecurityITest (251014 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:25 dist-test-slave-ncp9 krb5kdc[20053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:25 dist-test-slave-ncp9 krb5kdc[20053](info): set up 2 sockets
Aug 06 04:33:25 dist-test-slave-ncp9 krb5kdc[20053](info): commencing operation
krb5kdc: starting...
W20260806 04:33:26.725634 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:26 dist-test-slave-ncp9 krb5kdc[20053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:26.859846 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39479
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:26.982756 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:26.983101 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:26.983165 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:26.987445 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:26.987530 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:26.987557 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:26.987579 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:26.987600 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:26.993438 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:26.994704 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:26.995819 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:27.005275 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.005275 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.005317 20073 server_base.cc:1034] running on GCE node
W20260806 04:33:27.005275 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.006215 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:27.007498 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:27.008708 20073 hybrid_clock.cc:648] HybridClock initialized: now 1785990807008680 us; error 35 us; skew 500 ppm
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:27.011780 20073 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:27.012946 20073 webserver.cc:460] Webserver started at http://127.28.94.62:34339/ using document root <none> and password file <none>
I20260806 04:33:27.013595 20073 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:27.013675 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:27.013908 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:27.016441 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "e8142f7abe7f4b8dba878218456c2f52"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "916411bf3b714402a2cce5cdb421a12a"
server_key_iv: "b22f066d31c7744489be17e88b126bb7"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.017182 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "e8142f7abe7f4b8dba878218456c2f52"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "916411bf3b714402a2cce5cdb421a12a"
server_key_iv: "b22f066d31c7744489be17e88b126bb7"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.021621 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:33:27.024587 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.026029 20073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:27.026188 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "e8142f7abe7f4b8dba878218456c2f52"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "916411bf3b714402a2cce5cdb421a12a"
server_key_iv: "b22f066d31c7744489be17e88b126bb7"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.026319 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:27.065258 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:27.068527 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:27.068734 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:27.076164 20073 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39479
I20260806 04:33:27.076229 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39479 every 8 connection(s)
I20260806 04:33:27.077257 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:27.078693 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20073
I20260806 04:33:27.078826 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:27.079172 29048 external_mini_cluster.cc:1428] Setting key bb4e3b95115b6e2888e6cfe79e0b8b00
I20260806 04:33:27.081111 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:27.087551 20171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: Bootstrap starting.
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:27.090695 20171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:27.091645 20171 log.cc:824] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:27.094316 20171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: No bootstrap required, opened a new log
I20260806 04:33:27.094501 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.080868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.28.94.62:39479)
0806 04:33:27.081448 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.081465 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.081503 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:33:27.082203 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.082222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.082569 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.082898 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:27.083739 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.084758 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.085651 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.086150 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.086311 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.089344 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:27.089350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:27.089365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:27.089402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:27.091149 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.091750 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.091769 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.091776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.091915 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.092252 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.092255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.092257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.092794 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:27.093001 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.093323 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.093621 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":176,"threads_started":1}
I20260806 04:33:27.098156 20171 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } }
I20260806 04:33:27.098428 20171 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:27.098515 20171 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8142f7abe7f4b8dba878218456c2f52, State: Initialized, Role: FOLLOWER
I20260806 04:33:27.098963 20171 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } }
I20260806 04:33:27.099102 20171 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:27.099171 20171 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:27.099278 20171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:27.100366 20171 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } }
I20260806 04:33:27.100744 20171 leader_election.cc:302] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8142f7abe7f4b8dba878218456c2f52; no voters:
I20260806 04:33:27.101130 20171 leader_election.cc:288] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:27.101251 20179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:27.101491 20179 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [term 1 LEADER]: Becoming Leader. State: Replica: e8142f7abe7f4b8dba878218456c2f52, State: Running, Role: LEADER
I20260806 04:33:27.101887 20179 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } }
I20260806 04:33:27.102578 20171 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:27.103533 20181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8142f7abe7f4b8dba878218456c2f52. Latest consensus state: current_term: 1 leader_uuid: "e8142f7abe7f4b8dba878218456c2f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } } }
I20260806 04:33:27.103688 20181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:27.104208 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8142f7abe7f4b8dba878218456c2f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8142f7abe7f4b8dba878218456c2f52" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39479 } } }
I20260806 04:33:27.104286 20191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:27.105091 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:27.107851 20191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:27.113955 20191 catalog_manager.cc:1347] Generated new cluster ID: d332533093714d36982bc787a15bbc97
I20260806 04:33:27.114063 20191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:27.159027 20191 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:27.160370 20191 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:27.168292 20191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: Generated new TSK 0
I20260806 04:33:27.169212 20191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:33:27.203635 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:27.332261 20215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:27.332580 20215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:27.332643 20215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:27.336993 20215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:27.337075 20215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:27.337139 20215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:27.342875 20215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:27.344143 20215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:27.345250 20215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:33:27.355960 20215 server_base.cc:1034] running on GCE node
W20260806 04:33:27.355901 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.355903 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.355901 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.356700 20215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:27.357498 20215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:27.358708 20215 hybrid_clock.cc:648] HybridClock initialized: now 1785990807358684 us; error 37 us; skew 500 ppm
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:27.361769 20215 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:33:27.362802 20215 webserver.cc:460] Webserver started at http://127.28.94.1:34585/ using document root <none> and password file <none>
I20260806 04:33:27.363390 20215 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:27.363477 20215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:27.363711 20215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:27.365998 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "089026e4804d4f95b22295bde67fa8dc"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "b1cb3fba343f2a73c524df0b30eae90b"
server_key_iv: "d2f7914c793c55d82b5eb7df56df0e0d"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.366582 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "089026e4804d4f95b22295bde67fa8dc"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "b1cb3fba343f2a73c524df0b30eae90b"
server_key_iv: "d2f7914c793c55d82b5eb7df56df0e0d"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.370731 20215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:27.373495 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.374886 20215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:27.375011 20215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "089026e4804d4f95b22295bde67fa8dc"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "b1cb3fba343f2a73c524df0b30eae90b"
server_key_iv: "d2f7914c793c55d82b5eb7df56df0e0d"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.375121 20215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:27.386278 20215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:27.389741 20215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:27.389925 20215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:27.390530 20215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:27.391430 20215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:27.391491 20215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.391534 20215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:27.391556 20215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.400781 20215 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:41123
I20260806 04:33:27.400825 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:41123 every 8 connection(s)
I20260806 04:33:27.401877 20215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:27.404429 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20215
I20260806 04:33:27.404557 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:27.404924 29048 external_mini_cluster.cc:1428] Setting key 9be115901e150059ef0ef5211ac0c321
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:33:27.418299 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.403949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:43283 (local address 127.28.94.62:39479)
0806 04:33:27.404223 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.404228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.405251 (+ 1023us) server_negotiation.cc:407] Connection header received
0806 04:33:27.406957 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.406960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.407009 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.407077 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:27.408714 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.409325 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.410986 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.411253 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.411337 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.414489 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:27.414492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:27.414494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:27.414523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:27.415920 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.416472 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.416475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.416476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.416571 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.416982 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.416986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.416988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.417205 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:27.417296 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.417954 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.418117 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:33:27.419320 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.404411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39479 (local address 127.28.94.1:43283)
0806 04:33:27.405058 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.405126 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.406702 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.407294 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.407304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.407831 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:27.408555 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.408577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.409460 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.409464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.410836 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.410851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.411430 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.411434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.411664 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.412382 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:27.412405 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:27.414270 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:27.416056 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.416063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.416086 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.416365 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.416686 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.416689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.416691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.416852 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.417316 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:27.417324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:27.417670 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.417859 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.418247 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":167,"threads_started":1}
I20260806 04:33:27.420755 20389 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39479
I20260806 04:33:27.421085 20389 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:27.421782 20389 heartbeater.cc:511] Master 127.28.94.62:39479 requested a full tablet report, sending...
I20260806 04:33:27.423673 20116 ts_manager.cc:194] Registered new tserver with Master: 089026e4804d4f95b22295bde67fa8dc (127.28.94.1:41123)
I20260806 04:33:27.425354 20116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:43283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:33:27.437942 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:27.561789 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:27.562119 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:27.562163 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:27.566484 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:27.566565 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:27.566625 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:27.572288 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:27.573563 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:27.574642 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:27.585299 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.585506 20397 server_base.cc:1034] running on GCE node
W20260806 04:33:27.585299 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.585342 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.586055 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:27.586794 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:27.587960 20397 hybrid_clock.cc:648] HybridClock initialized: now 1785990807587926 us; error 53 us; skew 500 ppm
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:27.590934 20397 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:33:27.591974 20397 webserver.cc:460] Webserver started at http://127.28.94.2:41075/ using document root <none> and password file <none>
I20260806 04:33:27.592584 20397 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:27.592667 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:27.592911 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:27.595155 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "4b5207b9a81441a4b1e1a5a6217419b7"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "aeee5a68de90407793d5c4988f5df648"
server_key_iv: "074963f84475f3f3d3718bba81c7a337"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.595777 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "4b5207b9a81441a4b1e1a5a6217419b7"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "aeee5a68de90407793d5c4988f5df648"
server_key_iv: "074963f84475f3f3d3718bba81c7a337"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.599964 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:33:27.602926 20417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.604389 20397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:27.604535 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "4b5207b9a81441a4b1e1a5a6217419b7"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "aeee5a68de90407793d5c4988f5df648"
server_key_iv: "074963f84475f3f3d3718bba81c7a337"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.604665 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:27.615485 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:27.618806 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:27.619017 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:27.619699 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:27.620645 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:27.620698 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.620757 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:27.620786 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.630491 20397 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:43111
I20260806 04:33:27.630559 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:43111 every 8 connection(s)
I20260806 04:33:27.631554 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:27.636435 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20397
I20260806 04:33:27.636559 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:27.636832 29048 external_mini_cluster.cc:1428] Setting key 84c47042f4ba6a5db9ffeeb2a577dc62
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:27.646631 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.633603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:57005 (local address 127.28.94.62:39479)
0806 04:33:27.633803 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.633808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.634706 (+ 898us) server_negotiation.cc:407] Connection header received
0806 04:33:27.636040 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.636043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.636123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.636218 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:27.637590 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.638190 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.639442 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.639752 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.639860 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.642592 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:27.642597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:27.642600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:27.642633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:27.644210 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.644830 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.644835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.644836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.644969 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.645359 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.645362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.645364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.645633 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:27.645742 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.646293 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.646442 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 04:33:27.647505 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.634027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39479 (local address 127.28.94.2:57005)
0806 04:33:27.634536 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.634586 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.635813 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.636379 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.636387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.636820 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:27.637452 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.637468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.638322 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.638325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.639308 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.639319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.639893 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.639895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.640050 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.640664 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:27.640684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:27.642387 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:27.644368 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.644375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.644393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.644715 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.645079 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.645082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.645084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.645262 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.645732 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:27.645738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:27.646028 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.646213 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.646519 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:33:27.648862 20580 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39479
I20260806 04:33:27.649180 20580 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:27.649749 20580 heartbeater.cc:511] Master 127.28.94.62:39479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:33:27.651015 20116 ts_manager.cc:194] Registered new tserver with Master: 4b5207b9a81441a4b1e1a5a6217419b7 (127.28.94.2:43111)
I20260806 04:33:27.651782 20116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:57005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:33:27.667941 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:27.788264 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:27.788578 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:27.788625 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:27.792788 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:27.792866 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:27.792945 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:27.798593 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:27.799751 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:27.800797 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:33:27.811218 20585 server_base.cc:1034] running on GCE node
W20260806 04:33:27.811177 20598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.811149 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:27.811154 20597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:27.811918 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:27.812651 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:27.813809 20585 hybrid_clock.cc:648] HybridClock initialized: now 1785990807813781 us; error 42 us; skew 500 ppm
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:27.816681 20585 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:33:27.817728 20585 webserver.cc:460] Webserver started at http://127.28.94.3:37013/ using document root <none> and password file <none>
I20260806 04:33:27.818305 20585 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:27.818367 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:27.818557 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:27.820767 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "b91efb7c8aa444fcb5844fe8c216f616"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "7a3bcc790d759ceeced8909bee387f29"
server_key_iv: "fe0867d405fad94f4f9bac6e1bbe476a"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.821415 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "b91efb7c8aa444fcb5844fe8c216f616"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "7a3bcc790d759ceeced8909bee387f29"
server_key_iv: "fe0867d405fad94f4f9bac6e1bbe476a"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.825671 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:27.828398 20613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.829898 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:27.830037 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "b91efb7c8aa444fcb5844fe8c216f616"
format_stamp: "Formatted at 2026-08-06 04:33:27 on dist-test-slave-ncp9"
server_key: "7a3bcc790d759ceeced8909bee387f29"
server_key_iv: "fe0867d405fad94f4f9bac6e1bbe476a"
server_key_version: "encryptionkey@0"
I20260806 04:33:27.830163 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:27.850422 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:27.853964 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:27.854166 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:27.854830 20585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:27.855757 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:27.855810 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.855871 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:27.855901 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:27.864938 20585 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:40339
I20260806 04:33:27.864979 20768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:40339 every 8 connection(s)
I20260806 04:33:27.865950 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:27.866281 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20585
I20260806 04:33:27.866421 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:27.866781 29048 external_mini_cluster.cc:1428] Setting key 5011e653275fb6c4e4f2bab1c4125503
Aug 06 04:33:27 dist-test-slave-ncp9 krb5kdc[20053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990807, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:27.880481 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.867990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:49541 (local address 127.28.94.62:39479)
0806 04:33:27.868131 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.868135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.869309 (+ 1174us) server_negotiation.cc:407] Connection header received
0806 04:33:27.870659 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.870664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.870724 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.870797 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:27.872040 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.872568 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.873839 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.874067 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.874159 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.876679 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:27.876682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:27.876685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:27.876715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:27.878095 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.878653 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.878657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.878658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.878753 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.879170 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.879173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.879174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.879383 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:27.879475 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.880170 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.880310 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:33:27.881538 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.868534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39479 (local address 127.28.94.3:49541)
0806 04:33:27.869152 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.869215 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.870429 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.870994 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.871002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.871389 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:27.871896 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.871910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.872692 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.872696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.873715 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.873727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.874228 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.874231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.874404 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.874958 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:27.874974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:27.876479 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:27.878199 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.878204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.878221 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.878502 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.878897 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:27.878899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:27.878900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:27.879062 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:27.879493 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:27.879502 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:27.879875 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.880104 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.880473 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":164,"threads_started":1}
I20260806 04:33:27.882725 20769 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39479
I20260806 04:33:27.882987 20769 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:27.883563 20769 heartbeater.cc:511] Master 127.28.94.62:39479 requested a full tablet report, sending...
I20260806 04:33:27.884748 20116 ts_manager.cc:194] Registered new tserver with Master: b91efb7c8aa444fcb5844fe8c216f616 (127.28.94.3:40339)
I20260806 04:33:27.885365 20116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:49541
I20260806 04:33:27.893484 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:27.903092 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.895639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.28.94.62:39479)
0806 04:33:27.895802 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.895807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.895931 (+ 124us) server_negotiation.cc:407] Connection header received
0806 04:33:27.896068 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.896072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.896121 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.896200 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:27.896948 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.897498 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.898359 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.898581 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.898672 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.900079 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:27.900082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:27.900084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:27.900113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:27.901340 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.901759 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.901762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.901763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.901847 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:27.902098 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:27.902102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:27.902104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:27.902421 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:27.902549 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.902798 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.902922 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:33:27.906263 20116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:27.909529 20116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:27.924989 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.918978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:41123 (local address 127.0.0.1:48318)
0806 04:33:27.919517 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.919572 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.919748 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.920751 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.920762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.920794 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:27.921005 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.921012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922144 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.922148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.923554 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.923565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.924364 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.924368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.924480 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.924527 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.924646 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.924787 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":52,"spinlock_wait_cycles":3840,"thread_start_us":115,"threads_started":1}
I20260806 04:33:27.924986 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.919724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.28.94.1:41123)
0806 04:33:27.920295 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.920309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.920330 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:33:27.920401 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.920406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.920575 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.920770 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:27.921110 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922038 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.923666 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.924240 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.924344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.924508 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.924694 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.924799 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":96,"threads_started":1}
I20260806 04:33:27.925948 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.920162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.28.94.3:40339)
0806 04:33:27.920472 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.920484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.920864 (+ 380us) server_negotiation.cc:407] Connection header received
0806 04:33:27.920967 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.920972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.921125 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.921283 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:27.921483 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922697 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.924258 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.925118 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.925234 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.925378 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.925722 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.925819 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
I20260806 04:33:27.925937 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.919614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40339 (local address 127.0.0.1:35600)
0806 04:33:27.920753 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.920768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.920850 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.921271 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.921273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.921290 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:27.921372 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.921377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922854 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.922857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.924102 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.924114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.925445 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.925449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.925579 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.925599 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.925663 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.925772 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":29,"thread_start_us":106,"threads_started":1}
I20260806 04:33:27.925945 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.918842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:43111 (local address 127.0.0.1:47984)
0806 04:33:27.919517 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:27.919572 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:27.919987 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:27.920749 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:27.920755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:27.920797 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:27.921535 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.921542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922930 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.922933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.924116 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:27.924123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.925424 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.925427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:27.925591 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.925617 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:33:27.925687 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:33:27.925791 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":688,"spinlock_wait_cycles":677504,"thread_start_us":93,"threads_started":1}
I20260806 04:33:27.926265 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:27.918990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.28.94.2:43111)
0806 04:33:27.919882 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:27.919894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:27.919921 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:33:27.920149 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:27.920165 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:27.920392 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:27.920540 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:27.921671 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.922797 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.924242 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:27.925107 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:27.925219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:27.925355 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:27.926016 (+ 661us) server_negotiation.cc:299] Negotiation successful
0806 04:33:27.926118 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":113,"threads_started":1}
I20260806 04:33:27.928834 20296 tablet_service.cc:1467] Processing CreateTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d (DEFAULT_TABLE table=test-table [id=f0bd6d393afe4408a5338d9b53725d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:27.929260 20481 tablet_service.cc:1467] Processing CreateTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d (DEFAULT_TABLE table=test-table [id=f0bd6d393afe4408a5338d9b53725d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:27.929353 20675 tablet_service.cc:1467] Processing CreateTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d (DEFAULT_TABLE table=test-table [id=f0bd6d393afe4408a5338d9b53725d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:27.930218 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf83b691d6045a3b24ecb6a1af3982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:27.930236 20481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf83b691d6045a3b24ecb6a1af3982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:27.930333 20675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf83b691d6045a3b24ecb6a1af3982d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:27.937229 20789 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Bootstrap starting.
I20260806 04:33:27.938416 20790 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Bootstrap starting.
I20260806 04:33:27.938855 20791 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Bootstrap starting.
I20260806 04:33:27.939888 20789 tablet_bootstrap.cc:654] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:27.940328 20790 tablet_bootstrap.cc:654] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:27.941179 20790 log.cc:824] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:27.941239 20789 log.cc:824] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:27.941578 20791 tablet_bootstrap.cc:654] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:27.942533 20791 log.cc:824] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:27.943976 20790 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: No bootstrap required, opened a new log
I20260806 04:33:27.944227 20790 ts_tablet_manager.cc:1397] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:33:27.944291 20789 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: No bootstrap required, opened a new log
I20260806 04:33:27.944396 20791 tablet_bootstrap.cc:492] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: No bootstrap required, opened a new log
I20260806 04:33:27.944466 20789 ts_tablet_manager.cc:1397] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:33:27.944594 20791 ts_tablet_manager.cc:1397] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 04:33:27.948578 20791 raft_consensus.cc:348] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.948966 20791 raft_consensus.cc:374] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:27.948772 20790 raft_consensus.cc:348] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.949023 20791 raft_consensus.cc:729] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5207b9a81441a4b1e1a5a6217419b7, State: Initialized, Role: FOLLOWER
I20260806 04:33:27.948853 20789 raft_consensus.cc:348] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.949079 20790 raft_consensus.cc:374] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:27.949122 20789 raft_consensus.cc:374] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:27.949136 20790 raft_consensus.cc:729] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089026e4804d4f95b22295bde67fa8dc, State: Initialized, Role: FOLLOWER
I20260806 04:33:27.949167 20789 raft_consensus.cc:729] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91efb7c8aa444fcb5844fe8c216f616, State: Initialized, Role: FOLLOWER
I20260806 04:33:27.949589 20789 consensus_queue.cc:260] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.949585 20791 consensus_queue.cc:260] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.950174 20790 consensus_queue.cc:260] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:27.950613 20769 heartbeater.cc:503] Master 127.28.94.62:39479 was elected leader, sending a full tablet report...
I20260806 04:33:27.950810 20791 ts_tablet_manager.cc:1428] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:33:27.950811 20789 ts_tablet_manager.cc:1428] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:33:27.951198 20790 ts_tablet_manager.cc:1428] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:33:27.951323 20389 heartbeater.cc:503] Master 127.28.94.62:39479 was elected leader, sending a full tablet report...
I20260806 04:33:27.951838 20580 heartbeater.cc:503] Master 127.28.94.62:39479 was elected leader, sending a full tablet report...
W20260806 04:33:28.117250 20770 tablet.cc:2369] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:28.132965 20581 tablet.cc:2369] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:28.150349 20796 raft_consensus.cc:482] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:28.150597 20796 raft_consensus.cc:504] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:28.151881 20796 leader_election.cc:288] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 089026e4804d4f95b22295bde67fa8dc (127.28.94.1:41123), b91efb7c8aa444fcb5844fe8c216f616 (127.28.94.3:40339)
W20260806 04:33:28.153349 20392 tablet.cc:2369] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:28.156544 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.152310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:38309 (local address 127.28.94.3:40339)
0806 04:33:28.152477 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:28.152482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:28.152708 (+ 226us) server_negotiation.cc:407] Connection header received
0806 04:33:28.152818 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:28.152821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:28.152868 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:28.152973 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:28.153324 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.154166 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.155497 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.155929 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.156013 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.156088 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:28.156334 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 04:33:28.156412 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:33:28.156584 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.152322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:44957 (local address 127.28.94.1:41123)
0806 04:33:28.152478 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:28.152482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:28.152639 (+ 157us) server_negotiation.cc:407] Connection header received
0806 04:33:28.152805 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:28.152808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:28.152868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:28.152979 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:28.153423 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.154168 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.155494 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.155961 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.156080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.156156 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:28.156363 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 04:33:28.156445 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:33:28.156598 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.152144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:41123 (local address 127.28.94.2:44957)
0806 04:33:28.152523 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:28.152541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:28.152689 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:28.153104 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:28.153108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:28.153129 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:28.153284 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.153291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.154302 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.154305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.155350 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.155362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.156101 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.156104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.156233 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.156252 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:33:28.156325 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:33:28.156405 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":47,"thread_start_us":95,"threads_started":1}
I20260806 04:33:28.156607 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.152216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:40339 (local address 127.28.94.2:38309)
0806 04:33:28.152611 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:28.152626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:28.152721 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:28.153009 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:28.153012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:28.153037 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:28.153188 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.153195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.154342 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.154345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.155345 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.155357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.156060 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.156064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.156174 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.156221 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 04:33:28.156294 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 04:33:28.156388 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":5,"thread_start_us":74,"threads_started":1}
I20260806 04:33:28.157416 20710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf83b691d6045a3b24ecb6a1af3982d" candidate_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91efb7c8aa444fcb5844fe8c216f616" is_pre_election: true
I20260806 04:33:28.157423 20326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf83b691d6045a3b24ecb6a1af3982d" candidate_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089026e4804d4f95b22295bde67fa8dc" is_pre_election: true
I20260806 04:33:28.157737 20710 raft_consensus.cc:2454] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5207b9a81441a4b1e1a5a6217419b7 in term 0.
I20260806 04:33:28.157742 20326 raft_consensus.cc:2454] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5207b9a81441a4b1e1a5a6217419b7 in term 0.
I20260806 04:33:28.158231 20419 leader_election.cc:302] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5207b9a81441a4b1e1a5a6217419b7, b91efb7c8aa444fcb5844fe8c216f616; no voters:
I20260806 04:33:28.158533 20796 raft_consensus.cc:2781] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:28.158628 20796 raft_consensus.cc:482] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:28.158663 20796 raft_consensus.cc:3037] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:28.159763 20796 raft_consensus.cc:504] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:28.160154 20796 leader_election.cc:288] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [CANDIDATE]: Term 1 election: Requested vote from peers 089026e4804d4f95b22295bde67fa8dc (127.28.94.1:41123), b91efb7c8aa444fcb5844fe8c216f616 (127.28.94.3:40339)
I20260806 04:33:28.160588 20710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf83b691d6045a3b24ecb6a1af3982d" candidate_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91efb7c8aa444fcb5844fe8c216f616"
I20260806 04:33:28.160588 20326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf83b691d6045a3b24ecb6a1af3982d" candidate_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089026e4804d4f95b22295bde67fa8dc"
I20260806 04:33:28.160727 20326 raft_consensus.cc:3037] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:28.160718 20710 raft_consensus.cc:3037] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:28.161837 20710 raft_consensus.cc:2454] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5207b9a81441a4b1e1a5a6217419b7 in term 1.
I20260806 04:33:28.161854 20326 raft_consensus.cc:2454] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5207b9a81441a4b1e1a5a6217419b7 in term 1.
I20260806 04:33:28.162184 20419 leader_election.cc:302] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5207b9a81441a4b1e1a5a6217419b7, b91efb7c8aa444fcb5844fe8c216f616; no voters:
I20260806 04:33:28.162392 20796 raft_consensus.cc:2781] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:28.162633 20796 raft_consensus.cc:686] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4b5207b9a81441a4b1e1a5a6217419b7, State: Running, Role: LEADER
I20260806 04:33:28.162992 20796 consensus_queue.cc:237] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } }
I20260806 04:33:28.166661 20116 catalog_manager.cc:5495] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5207b9a81441a4b1e1a5a6217419b7 (127.28.94.2). New cstate: current_term: 1 leader_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5207b9a81441a4b1e1a5a6217419b7" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 43111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 } health_report { overall_health: UNKNOWN } } }
I20260806 04:33:28.220242 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.216643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.28.94.2:43111)
0806 04:33:28.216787 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:28.216791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:28.216818 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:33:28.216948 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:28.216951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:28.217017 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:28.217115 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:28.217373 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.217886 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.218734 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.218952 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.219042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.219362 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:28.219827 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:28.219940 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:28.220010 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 04:33:28.220077 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:33:28.226969 20710 raft_consensus.cc:1261] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 1 FOLLOWER]: Refusing update from remote peer 4b5207b9a81441a4b1e1a5a6217419b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:28.226969 20326 raft_consensus.cc:1261] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 1 FOLLOWER]: Refusing update from remote peer 4b5207b9a81441a4b1e1a5a6217419b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:28.227784 20806 consensus_queue.cc:1035] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "089026e4804d4f95b22295bde67fa8dc" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:28.227998 20796 consensus_queue.cc:1035] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91efb7c8aa444fcb5844fe8c216f616" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:28.237668 20811 mvcc.cc:205] Tried to move back new op lower bound from 7315418350489583616 to 7315418350238437376. Current Snapshot: MvccSnapshot[applied={T|T < 7315418350489583616}]
I20260806 04:33:28.246466 20115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:28.246711 20115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:28.249743 20115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8142f7abe7f4b8dba878218456c2f52: Sending DeleteTablet for 3 replicas of tablet fcf83b691d6045a3b24ecb6a1af3982d
I20260806 04:33:28.250813 20481 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:47984
I20260806 04:33:28.250818 20675 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:35600
I20260806 04:33:28.250838 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet fcf83b691d6045a3b24ecb6a1af3982d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:28 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:48318
I20260806 04:33:28.251442 20826 tablet_replica.cc:330] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc: stopping tablet replica
I20260806 04:33:28.251860 20826 raft_consensus.cc:2229] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:28.252123 20828 tablet_replica.cc:330] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: stopping tablet replica
I20260806 04:33:28.252274 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20215
I20260806 04:33:28.252271 20826 raft_consensus.cc:2258] T fcf83b691d6045a3b24ecb6a1af3982d P 089026e4804d4f95b22295bde67fa8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:28.252419 20828 raft_consensus.cc:2229] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:28.252856 20828 raft_consensus.cc:2258] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:28.253459 20827 tablet_replica.cc:330] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: stopping tablet replica
I20260806 04:33:28.253835 20827 raft_consensus.cc:2229] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:28.254206 20827 raft_consensus.cc:2258] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:28.254546 20828 ts_tablet_manager.cc:1905] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:28.256340 20827 ts_tablet_manager.cc:1905] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:28.257699 20828 ts_tablet_manager.cc:1918] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:28.257902 20828 log.cc:1196] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/fcf83b691d6045a3b24ecb6a1af3982d
I20260806 04:33:28.258359 20828 ts_tablet_manager.cc:1939] T fcf83b691d6045a3b24ecb6a1af3982d P 4b5207b9a81441a4b1e1a5a6217419b7: Deleting consensus metadata
I20260806 04:33:28.259087 20827 ts_tablet_manager.cc:1918] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:28.259263 20827 log.cc:1196] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/fcf83b691d6045a3b24ecb6a1af3982d
I20260806 04:33:28.259665 20827 ts_tablet_manager.cc:1939] T fcf83b691d6045a3b24ecb6a1af3982d P b91efb7c8aa444fcb5844fe8c216f616: Deleting consensus metadata
I20260806 04:33:28.260030 20096 catalog_manager.cc:4841] TS 4b5207b9a81441a4b1e1a5a6217419b7 (127.28.94.2:43111): tablet fcf83b691d6045a3b24ecb6a1af3982d (table test-table [id=f0bd6d393afe4408a5338d9b53725d8b]) successfully deleted
I20260806 04:33:28.261129 20096 catalog_manager.cc:4841] TS b91efb7c8aa444fcb5844fe8c216f616 (127.28.94.3:40339): tablet fcf83b691d6045a3b24ecb6a1af3982d (table test-table [id=f0bd6d393afe4408a5338d9b53725d8b]) successfully deleted
I20260806 04:33:28.277093 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20397
W20260806 04:33:28.277107 20099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:41123 (error 108)
I20260806 04:33:28.281515 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.281125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:41123 (local address 127.0.0.1:48334)
0806 04:33:28.281333 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:28.281409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260806 04:33:28.281917 20099 catalog_manager.cc:4584] TS 089026e4804d4f95b22295bde67fa8dc (127.28.94.1:41123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcf83b691d6045a3b24ecb6a1af3982d: Network error: Client connection negotiation failed: client connection to 127.28.94.1:41123: connect: Connection refused (error 111)
I20260806 04:33:28.289165 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20585
I20260806 04:33:28.305804 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20073
2026-08-06T04:33:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2712 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T04:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:28Z Disabled control of system clock
I20260806 04:33:28.346534 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:35543
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:28.467056 20836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:28.467410 20836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:28.467463 20836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:28.471782 20836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:28.471884 20836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:28.471906 20836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:28.471928 20836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:28.471947 20836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:28.477206 20836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:35543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:28.478528 20836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:28.480079 20836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:28.489308 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:28.489308 20843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:28.489310 20842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:28.489670 20836 server_base.cc:1034] running on GCE node
I20260806 04:33:28.490398 20836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:28.491691 20836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:28.492866 20836 hybrid_clock.cc:648] HybridClock initialized: now 1785990808492850 us; error 32 us; skew 500 ppm
I20260806 04:33:28.494971 20836 webserver.cc:460] Webserver started at http://127.28.94.62:46425/ using document root <none> and password file <none>
I20260806 04:33:28.495613 20836 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:28.495692 20836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:28.495926 20836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:28.498250 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "55f7a72e6d3b4dfa88f207911205d10b"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "de2e6c45f3851df2ac39b09219d8eca9"
server_key_iv: "efc422166499e71388107c3e015cfb5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.498862 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "55f7a72e6d3b4dfa88f207911205d10b"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "de2e6c45f3851df2ac39b09219d8eca9"
server_key_iv: "efc422166499e71388107c3e015cfb5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.503209 20836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:28.506021 20857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:28.507406 20836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:28.507570 20836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "55f7a72e6d3b4dfa88f207911205d10b"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "de2e6c45f3851df2ac39b09219d8eca9"
server_key_iv: "efc422166499e71388107c3e015cfb5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.507694 20836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:28.524665 20836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:28.525516 20836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:28.525722 20836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:28.532996 20836 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:35543
I20260806 04:33:28.533010 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:35543 every 8 connection(s)
I20260806 04:33:28.534060 20836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:28.537663 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:28.544037 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20836
I20260806 04:33:28.544081 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: Bootstrap starting.
I20260806 04:33:28.544200 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:28.544520 29048 external_mini_cluster.cc:1428] Setting key f404466fd9af37d886139ab833f2c683
I20260806 04:33:28.547139 20933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:28.547966 20933 log.cc:824] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:28.550060 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: No bootstrap required, opened a new log
I20260806 04:33:28.553881 20933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } }
I20260806 04:33:28.554177 20933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:28.554236 20933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f7a72e6d3b4dfa88f207911205d10b, State: Initialized, Role: FOLLOWER
I20260806 04:33:28.554517 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.546086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.28.94.62:35543)
0806 04:33:28.546862 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:28.546879 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:28.546918 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:33:28.547666 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:28.547689 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:28.548160 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:28.548545 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:28.549065 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.550042 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.550957 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.551460 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.551616 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.552465 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:28.552470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:28.552485 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:28.552955 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:28.553019 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:28.553042 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:28.553138 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:28.553281 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 04:33:28.553583 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":163,"threads_started":1}
I20260806 04:33:28.554764 20933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } }
I20260806 04:33:28.554936 20933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:28.555006 20933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:28.555106 20933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:28.556226 20933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } }
I20260806 04:33:28.556591 20933 leader_election.cc:302] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f7a72e6d3b4dfa88f207911205d10b; no voters:
I20260806 04:33:28.556947 20933 leader_election.cc:288] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:28.557060 20938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:28.557291 20938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [term 1 LEADER]: Becoming Leader. State: Replica: 55f7a72e6d3b4dfa88f207911205d10b, State: Running, Role: LEADER
I20260806 04:33:28.557665 20938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } }
I20260806 04:33:28.558245 20933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:28.559371 20940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f7a72e6d3b4dfa88f207911205d10b. Latest consensus state: current_term: 1 leader_uuid: "55f7a72e6d3b4dfa88f207911205d10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } } }
I20260806 04:33:28.559511 20940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:28.559794 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f7a72e6d3b4dfa88f207911205d10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f7a72e6d3b4dfa88f207911205d10b" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 35543 } } }
I20260806 04:33:28.559882 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:28.559922 20948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:28.563184 20948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:28.569671 20948 catalog_manager.cc:1347] Generated new cluster ID: 1d0c3db3a6ad43b9b48ab30b8d510316
I20260806 04:33:28.569818 20948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:28.586953 20948 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:28.588382 20948 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:28.596431 20948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: Generated new TSK 0
I20260806 04:33:28.597397 20948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:28.610927 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--builtin_ntp_servers=127.28.94.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:28.734506 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:28.734844 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:28.734885 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:28.739079 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:28.739156 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:28.739214 20959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:28.744277 20959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:28.745534 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:28.747035 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:28.757436 20966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:28.757436 20967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:28.757594 20959 server_base.cc:1034] running on GCE node
W20260806 04:33:28.757444 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:28.758296 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:28.759095 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:28.760272 20959 hybrid_clock.cc:648] HybridClock initialized: now 1785990808760246 us; error 36 us; skew 500 ppm
I20260806 04:33:28.762395 20959 webserver.cc:460] Webserver started at http://127.28.94.1:34755/ using document root <none> and password file <none>
I20260806 04:33:28.763052 20959 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:28.763136 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:28.763365 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:28.765760 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "870a8d60724f4bb69296e0a8b8e4469a"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "456d11605f439310eaa6513e5996e82e"
server_key_iv: "c86dbf03530428c9e7820159155e7449"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.766379 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "870a8d60724f4bb69296e0a8b8e4469a"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "456d11605f439310eaa6513e5996e82e"
server_key_iv: "c86dbf03530428c9e7820159155e7449"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.770428 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:28.773254 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:28.774593 20959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:33:28.774734 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "870a8d60724f4bb69296e0a8b8e4469a"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "456d11605f439310eaa6513e5996e82e"
server_key_iv: "c86dbf03530428c9e7820159155e7449"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.774870 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:28.805853 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:28.806661 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:28.806864 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:28.807528 20959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:28.808621 20959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:28.808682 20959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:28.808739 20959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:28.808766 20959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:28.818068 20959 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44615
I20260806 04:33:28.818089 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44615 every 8 connection(s)
I20260806 04:33:28.819082 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:28.819490 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 20959
I20260806 04:33:28.819609 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:28.819941 29048 external_mini_cluster.cc:1428] Setting key 6f473b4a7569b93ac08c7b1473bcc204
I20260806 04:33:28.822361 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--builtin_ntp_servers=127.28.94.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:28.829352 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.821100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:41043 (local address 127.28.94.62:35543)
0806 04:33:28.821249 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:28.821254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:28.822724 (+ 1470us) server_negotiation.cc:407] Connection header received
0806 04:33:28.824262 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:28.824266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:28.824330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:28.824406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:28.825489 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.826036 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.827260 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.827486 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.827583 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.828793 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:28.828797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:28.828799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:28.828815 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:28.828835 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:28.828843 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:28.828942 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:28.829092 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 04:33:28.829188 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:33:28.830325 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:28.821707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35543 (local address 127.28.94.1:41043)
0806 04:33:28.822547 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:28.822611 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:28.823995 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:28.824623 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:28.824634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:28.824730 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:28.825339 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.825354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.826177 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.826180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.827117 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:28.827137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:28.827661 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:28.827665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:28.827852 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:28.828482 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:28.828484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:28.828533 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:28.828618 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:28.828942 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:28.828954 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:28.828962 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:33:28.829087 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 04:33:28.829439 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":167,"threads_started":1}
I20260806 04:33:28.831683 21132 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35543
I20260806 04:33:28.831946 21132 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:28.832561 21132 heartbeater.cc:511] Master 127.28.94.62:35543 requested a full tablet report, sending...
I20260806 04:33:28.834441 20879 ts_manager.cc:194] Registered new tserver with Master: 870a8d60724f4bb69296e0a8b8e4469a (127.28.94.1:44615)
I20260806 04:33:28.836092 20879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.1:41043
W20260806 04:33:28.945364 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:28.945680 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:28.945721 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:28.949919 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:28.949991 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:28.950048 21138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:28.955058 21138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:28.956205 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:28.957666 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:28.968245 21149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:28.968245 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:28.968284 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:28.968623 21138 server_base.cc:1034] running on GCE node
I20260806 04:33:28.969106 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:28.969820 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:28.970983 21138 hybrid_clock.cc:648] HybridClock initialized: now 1785990808970956 us; error 40 us; skew 500 ppm
I20260806 04:33:28.972954 21138 webserver.cc:460] Webserver started at http://127.28.94.2:41803/ using document root <none> and password file <none>
I20260806 04:33:28.973587 21138 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:28.973659 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:28.973862 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:28.976044 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "58e7dcc5d8d74c5ba78273b9485c1129"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "48799a03a38cc622d941457f3ef2c8eb"
server_key_iv: "fddec3a4dd001c5ed6531d485ffdaf92"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.976636 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "58e7dcc5d8d74c5ba78273b9485c1129"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "48799a03a38cc622d941457f3ef2c8eb"
server_key_iv: "fddec3a4dd001c5ed6531d485ffdaf92"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.981702 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 04:33:28.984390 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:28.985797 21138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:33:28.985935 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "58e7dcc5d8d74c5ba78273b9485c1129"
format_stamp: "Formatted at 2026-08-06 04:33:28 on dist-test-slave-ncp9"
server_key: "48799a03a38cc622d941457f3ef2c8eb"
server_key_iv: "fddec3a4dd001c5ed6531d485ffdaf92"
server_key_version: "encryptionkey@0"
I20260806 04:33:28.986057 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:29.010808 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:29.011659 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:29.011880 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:29.012539 21138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:29.013631 21138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:29.013684 21138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:29.013746 21138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:29.013773 21138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:29.022732 21138 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:42945
I20260806 04:33:29.022800 21320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:42945 every 8 connection(s)
I20260806 04:33:29.023739 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:29.030874 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 21138
I20260806 04:33:29.030998 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:29.031312 29048 external_mini_cluster.cc:1428] Setting key 6253b02989a6ec08f36b6f5514d8e2c1
I20260806 04:33:29.033634 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--builtin_ntp_servers=127.28.94.20:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:29.033696 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.025716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:53955 (local address 127.28.94.62:35543)
0806 04:33:29.025873 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.025878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.027089 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 04:33:29.028507 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.028513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.028590 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.028669 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:29.029698 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.030229 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.031519 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.031821 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.031918 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.033199 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:29.033202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:29.033204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:29.033217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:29.033239 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:29.033248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:29.033301 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.033482 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.033574 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:33:29.034969 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.026188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35543 (local address 127.28.94.2:53955)
0806 04:33:29.026917 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.026964 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.028287 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.028839 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.028846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.028995 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:29.029554 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.029569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.030325 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.030328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.031334 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.031346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.031998 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.032001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.032171 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.032807 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:29.032809 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:29.032872 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:29.033006 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:29.033324 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:29.033335 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:29.033343 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.033490 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.033861 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260806 04:33:29.036294 21321 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35543
I20260806 04:33:29.036619 21321 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:29.037367 21321 heartbeater.cc:511] Master 127.28.94.62:35543 requested a full tablet report, sending...
I20260806 04:33:29.038647 20879 ts_manager.cc:194] Registered new tserver with Master: 58e7dcc5d8d74c5ba78273b9485c1129 (127.28.94.2:42945)
I20260806 04:33:29.039409 20879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.2:53955
W20260806 04:33:29.158447 21326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:29.158758 21326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:29.158797 21326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:29.163142 21326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:29.163224 21326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:29.163282 21326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:29.168396 21326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:35543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:29.169682 21326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:29.171155 21326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:33:29.181603 21326 server_base.cc:1034] running on GCE node
W20260806 04:33:29.181569 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:29.181560 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:29.181859 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:29.182365 21326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:29.183117 21326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:29.184315 21326 hybrid_clock.cc:648] HybridClock initialized: now 1785990809184291 us; error 35 us; skew 500 ppm
I20260806 04:33:29.186417 21326 webserver.cc:460] Webserver started at http://127.28.94.3:38081/ using document root <none> and password file <none>
I20260806 04:33:29.187047 21326 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:29.187117 21326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:29.187340 21326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:29.189821 21326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "bfa487a3e79b4c699e9434c7c88138f6"
format_stamp: "Formatted at 2026-08-06 04:33:29 on dist-test-slave-ncp9"
server_key: "db47f8b11f77bacb795dced8b88e62b7"
server_key_iv: "fdf174477625e6b35316166e33662d81"
server_key_version: "encryptionkey@0"
I20260806 04:33:29.190444 21326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "bfa487a3e79b4c699e9434c7c88138f6"
format_stamp: "Formatted at 2026-08-06 04:33:29 on dist-test-slave-ncp9"
server_key: "db47f8b11f77bacb795dced8b88e62b7"
server_key_iv: "fdf174477625e6b35316166e33662d81"
server_key_version: "encryptionkey@0"
I20260806 04:33:29.194516 21326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:29.197232 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:29.198583 21326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:33:29.198742 21326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "bfa487a3e79b4c699e9434c7c88138f6"
format_stamp: "Formatted at 2026-08-06 04:33:29 on dist-test-slave-ncp9"
server_key: "db47f8b11f77bacb795dced8b88e62b7"
server_key_iv: "fdf174477625e6b35316166e33662d81"
server_key_version: "encryptionkey@0"
I20260806 04:33:29.198870 21326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:29.213541 21326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:29.214325 21326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:29.214521 21326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:29.215178 21326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:29.216252 21326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:29.216305 21326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:29.216349 21326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:29.216379 21326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:29.225440 21326 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:46761
I20260806 04:33:29.225471 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:46761 every 8 connection(s)
I20260806 04:33:29.226454 21326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:29.233361 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 21326
I20260806 04:33:29.233469 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:29.233716 29048 external_mini_cluster.cc:1428] Setting key f16dd29b355d90e15377e4f292a4489d
I20260806 04:33:29.236505 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.228479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:52441 (local address 127.28.94.62:35543)
0806 04:33:29.228625 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.228629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.229840 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 04:33:29.231320 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.231323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.231393 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.231469 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:29.232473 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.233064 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.234446 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.234737 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.234847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.236088 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:29.236091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:29.236093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:29.236105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:29.236126 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:29.236134 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:29.236184 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.236326 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.236383 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:33:29.237526 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.228932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:35543 (local address 127.28.94.3:52441)
0806 04:33:29.229666 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.229714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.231067 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.231684 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.231692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.231787 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:29.232314 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.232328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.233208 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.233211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.234327 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.234340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.234901 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.234904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.235119 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.235778 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:29.235780 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:29.235840 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:29.235934 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:29.236203 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:29.236211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:29.236215 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.236325 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.236597 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":121,"threads_started":1}
I20260806 04:33:29.238657 21524 heartbeater.cc:347] Connected to a master server at 127.28.94.62:35543
I20260806 04:33:29.238915 21524 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:29.239506 21524 heartbeater.cc:511] Master 127.28.94.62:35543 requested a full tablet report, sending...
I20260806 04:33:29.240664 20879 ts_manager.cc:194] Registered new tserver with Master: bfa487a3e79b4c699e9434c7c88138f6 (127.28.94.3:46761)
I20260806 04:33:29.241308 20879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.3:52441
I20260806 04:33:29.248054 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:29.253933 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.250149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.28.94.62:35543)
0806 04:33:29.250303 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.250309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.250451 (+ 142us) server_negotiation.cc:407] Connection header received
0806 04:33:29.250602 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.250605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.250650 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.250720 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:29.251004 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.251527 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.252381 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.252604 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.252695 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.253507 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:29.253511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:29.253513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:29.253527 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:29.253553 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:29.253561 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:29.253623 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.253710 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.253776 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:33:29.257033 20879 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:29.260294 20879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:29.276413 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.270598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.28.94.3:46761)
0806 04:33:29.271221 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.271234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.271254 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:33:29.271329 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.271334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.271548 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.271707 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:29.272324 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273321 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.275125 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.275678 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.275778 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.276000 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.276106 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.276227 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":202,"threads_started":1}
I20260806 04:33:29.276620 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.269758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44615 (local address 127.0.0.1:56574)
0806 04:33:29.270409 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.270464 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.270594 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.271735 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.271738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.271781 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:29.271992 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.272002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273540 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.273544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.274960 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.274972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.276060 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.276064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.276159 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.276220 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.276286 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.276427 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":164,"spinlock_wait_cycles":8832,"thread_start_us":91,"threads_started":1}
I20260806 04:33:29.276744 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.270882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.28.94.1:44615)
0806 04:33:29.271302 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.271316 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.271343 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:33:29.271415 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.271419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.271582 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.271894 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:29.272127 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273380 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.275094 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.275890 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.276225 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.276424 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.276482 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.276578 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260806 04:33:29.277379 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.269647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42945 (local address 127.0.0.1:59430)
0806 04:33:29.271028 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.271099 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.271221 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.272332 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.272335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.272355 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:29.272463 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.272470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273926 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.273931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.274795 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.274805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.276155 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.276159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.277029 (+ 870us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.277060 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.277148 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.277226 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260806 04:33:29.277581 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.270428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:46761 (local address 127.0.0.1:36620)
0806 04:33:29.270847 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.270861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.270943 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.271730 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.271737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.271780 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:29.272211 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.272219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273449 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.273452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.274960 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.274972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.275801 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.275804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.275920 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.275961 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.276076 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.277378 (+ 1302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":61,"spinlock_wait_cycles":6912,"thread_start_us":98,"threads_started":1}
I20260806 04:33:29.277772 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.269800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.28.94.2:42945)
0806 04:33:29.270978 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.270991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.271231 (+ 240us) server_negotiation.cc:407] Connection header received
0806 04:33:29.271329 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.271334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.271548 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.271707 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:29.272644 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.273806 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.274930 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.275760 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.275883 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.276093 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.277477 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.277593 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":109,"threads_started":1}
I20260806 04:33:29.280566 21219 tablet_service.cc:1467] Processing CreateTablet for tablet 53bec4aed1664349bac198e0d2a2394a (DEFAULT_TABLE table=test-table [id=aadcf02f9c674e049723280ee6a38c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:29.280579 21043 tablet_service.cc:1467] Processing CreateTablet for tablet 53bec4aed1664349bac198e0d2a2394a (DEFAULT_TABLE table=test-table [id=aadcf02f9c674e049723280ee6a38c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:29.280579 21421 tablet_service.cc:1467] Processing CreateTablet for tablet 53bec4aed1664349bac198e0d2a2394a (DEFAULT_TABLE table=test-table [id=aadcf02f9c674e049723280ee6a38c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:29.281725 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bec4aed1664349bac198e0d2a2394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:29.281739 21043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bec4aed1664349bac198e0d2a2394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:29.281739 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bec4aed1664349bac198e0d2a2394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:29.289140 21546 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Bootstrap starting.
I20260806 04:33:29.290410 21548 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Bootstrap starting.
I20260806 04:33:29.290827 21547 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Bootstrap starting.
I20260806 04:33:29.292424 21546 tablet_bootstrap.cc:654] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:29.293071 21548 tablet_bootstrap.cc:654] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:29.293071 21547 tablet_bootstrap.cc:654] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:29.293392 21546 log.cc:824] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:29.294078 21548 log.cc:824] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:29.294117 21547 log.cc:824] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:29.295660 21546 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: No bootstrap required, opened a new log
I20260806 04:33:29.295899 21546 ts_tablet_manager.cc:1397] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:33:29.296057 21547 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: No bootstrap required, opened a new log
I20260806 04:33:29.296221 21548 tablet_bootstrap.cc:492] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: No bootstrap required, opened a new log
I20260806 04:33:29.296312 21547 ts_tablet_manager.cc:1397] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260806 04:33:29.296466 21548 ts_tablet_manager.cc:1397] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:33:29.299957 21547 raft_consensus.cc:348] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.300257 21547 raft_consensus.cc:374] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:29.300315 21547 raft_consensus.cc:729] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa487a3e79b4c699e9434c7c88138f6, State: Initialized, Role: FOLLOWER
I20260806 04:33:29.300707 21548 raft_consensus.cc:348] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.300707 21546 raft_consensus.cc:348] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.300855 21547 consensus_queue.cc:260] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.301015 21546 raft_consensus.cc:374] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:29.301015 21548 raft_consensus.cc:374] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:29.301075 21548 raft_consensus.cc:729] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870a8d60724f4bb69296e0a8b8e4469a, State: Initialized, Role: FOLLOWER
I20260806 04:33:29.301106 21546 raft_consensus.cc:729] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e7dcc5d8d74c5ba78273b9485c1129, State: Initialized, Role: FOLLOWER
I20260806 04:33:29.301599 21546 consensus_queue.cc:260] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.301599 21548 consensus_queue.cc:260] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.301708 21524 heartbeater.cc:503] Master 127.28.94.62:35543 was elected leader, sending a full tablet report...
I20260806 04:33:29.302119 21547 ts_tablet_manager.cc:1428] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260806 04:33:29.302615 21546 ts_tablet_manager.cc:1428] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:33:29.302615 21548 ts_tablet_manager.cc:1428] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:33:29.302976 21321 heartbeater.cc:503] Master 127.28.94.62:35543 was elected leader, sending a full tablet report...
I20260806 04:33:29.303020 21132 heartbeater.cc:503] Master 127.28.94.62:35543 was elected leader, sending a full tablet report...
W20260806 04:33:29.320411 21133 tablet.cc:2369] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:29.427927 21554 raft_consensus.cc:482] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:29.428138 21554 raft_consensus.cc:504] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.429301 21554 leader_election.cc:288] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e7dcc5d8d74c5ba78273b9485c1129 (127.28.94.2:42945), 870a8d60724f4bb69296e0a8b8e4469a (127.28.94.1:44615)
I20260806 04:33:29.433295 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.429702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:49025 (local address 127.28.94.2:42945)
0806 04:33:29.429819 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.429824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.429840 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:33:29.429904 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.429907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.429962 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.430051 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:29.430460 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.431046 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432188 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.432597 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.432756 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.433072 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.433148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:33:29.433297 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.429687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:35023 (local address 127.28.94.1:44615)
0806 04:33:29.429818 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.429823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.430175 (+ 352us) server_negotiation.cc:407] Connection header received
0806 04:33:29.430306 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.430310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.430369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.430450 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:29.430696 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.431232 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432241 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.432676 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432793 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.432864 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.433077 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.433152 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:33:29.433311 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.429571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:42945 (local address 127.28.94.3:49025)
0806 04:33:29.429690 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.429705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.429808 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.430103 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.430106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.430132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:29.430316 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.430325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.431162 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.431166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.432042 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.432054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.432730 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.432915 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.432961 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.433036 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.433142 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":31,"spinlock_wait_cycles":26496}
I20260806 04:33:29.433326 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.429571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44615 (local address 127.28.94.3:35023)
0806 04:33:29.430056 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.430071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:29.430202 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:29.430460 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:29.430463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:29.430478 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:29.430594 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.430600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.431353 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.431356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.432102 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:29.432111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.432818 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.432821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:29.432958 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.432976 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 04:33:29.433042 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:33:29.433142 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":162,"spinlock_wait_cycles":34432,"thread_start_us":117,"threads_started":1}
I20260806 04:33:29.434188 21073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bec4aed1664349bac198e0d2a2394a" candidate_uuid: "bfa487a3e79b4c699e9434c7c88138f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870a8d60724f4bb69296e0a8b8e4469a" is_pre_election: true
I20260806 04:33:29.434183 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bec4aed1664349bac198e0d2a2394a" candidate_uuid: "bfa487a3e79b4c699e9434c7c88138f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" is_pre_election: true
I20260806 04:33:29.434516 21073 raft_consensus.cc:2454] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa487a3e79b4c699e9434c7c88138f6 in term 0.
I20260806 04:33:29.434516 21246 raft_consensus.cc:2454] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa487a3e79b4c699e9434c7c88138f6 in term 0.
I20260806 04:33:29.435043 21350 leader_election.cc:302] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870a8d60724f4bb69296e0a8b8e4469a, bfa487a3e79b4c699e9434c7c88138f6; no voters:
I20260806 04:33:29.435339 21554 raft_consensus.cc:2781] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:29.435433 21554 raft_consensus.cc:482] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:29.435483 21554 raft_consensus.cc:3037] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:29.436529 21554 raft_consensus.cc:504] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.436926 21554 leader_election.cc:288] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [CANDIDATE]: Term 1 election: Requested vote from peers 58e7dcc5d8d74c5ba78273b9485c1129 (127.28.94.2:42945), 870a8d60724f4bb69296e0a8b8e4469a (127.28.94.1:44615)
I20260806 04:33:29.437367 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bec4aed1664349bac198e0d2a2394a" candidate_uuid: "bfa487a3e79b4c699e9434c7c88138f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e7dcc5d8d74c5ba78273b9485c1129"
I20260806 04:33:29.437508 21246 raft_consensus.cc:3037] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:29.438650 21246 raft_consensus.cc:2454] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa487a3e79b4c699e9434c7c88138f6 in term 1.
I20260806 04:33:29.439003 21073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bec4aed1664349bac198e0d2a2394a" candidate_uuid: "bfa487a3e79b4c699e9434c7c88138f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870a8d60724f4bb69296e0a8b8e4469a"
I20260806 04:33:29.439038 21349 leader_election.cc:302] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e7dcc5d8d74c5ba78273b9485c1129, bfa487a3e79b4c699e9434c7c88138f6; no voters:
I20260806 04:33:29.439163 21073 raft_consensus.cc:3037] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:29.439229 21554 raft_consensus.cc:2781] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:29.439453 21554 raft_consensus.cc:686] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 1 LEADER]: Becoming Leader. State: Replica: bfa487a3e79b4c699e9434c7c88138f6, State: Running, Role: LEADER
I20260806 04:33:29.439867 21554 consensus_queue.cc:237] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } }
I20260806 04:33:29.440428 21073 raft_consensus.cc:2454] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa487a3e79b4c699e9434c7c88138f6 in term 1.
I20260806 04:33:29.443549 20875 catalog_manager.cc:5495] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa487a3e79b4c699e9434c7c88138f6 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "bfa487a3e79b4c699e9434c7c88138f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa487a3e79b4c699e9434c7c88138f6" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 46761 } health_report { overall_health: HEALTHY } } }
I20260806 04:33:29.471611 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.468383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.28.94.3:46761)
0806 04:33:29.468502 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:29.468506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:29.468520 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:33:29.468625 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:29.468628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:29.468697 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:29.468786 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:29.469007 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.469500 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.470308 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:29.470515 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:29.470583 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:29.470836 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:29.471294 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:29.471385 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:29.471434 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 04:33:29.471486 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 04:33:29.477443 21525 tablet.cc:2369] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:29.478395 21073 raft_consensus.cc:1261] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a [term 1 FOLLOWER]: Refusing update from remote peer bfa487a3e79b4c699e9434c7c88138f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:29.478397 21246 raft_consensus.cc:1261] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 1 FOLLOWER]: Refusing update from remote peer bfa487a3e79b4c699e9434c7c88138f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:29.479311 21561 consensus_queue.cc:1035] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870a8d60724f4bb69296e0a8b8e4469a" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:29.479540 21554 consensus_queue.cc:1035] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e7dcc5d8d74c5ba78273b9485c1129" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:29.488157 21564 mvcc.cc:205] Tried to move back new op lower bound from 7315418355614851072 to 7315418355468881920. Current Snapshot: MvccSnapshot[applied={T|T < 7315418355614851072}]
I20260806 04:33:29.490473 21566 mvcc.cc:205] Tried to move back new op lower bound from 7315418355614851072 to 7315418355468881920. Current Snapshot: MvccSnapshot[applied={T|T < 7315418355614851072 or (T in {7315418355614851072})}]
I20260806 04:33:29.499459 20875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:29.499655 20875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:29.502571 20875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55f7a72e6d3b4dfa88f207911205d10b: Sending DeleteTablet for 3 replicas of tablet 53bec4aed1664349bac198e0d2a2394a
I20260806 04:33:29.503489 21219 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bec4aed1664349bac198e0d2a2394a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:29 UTC) from {username='slave'} at 127.0.0.1:59430
I20260806 04:33:29.503676 21043 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bec4aed1664349bac198e0d2a2394a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:29 UTC) from {username='slave'} at 127.0.0.1:56574
I20260806 04:33:29.503710 21421 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bec4aed1664349bac198e0d2a2394a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:29 UTC) from {username='slave'} at 127.0.0.1:36620
I20260806 04:33:29.504165 21576 tablet_replica.cc:330] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: stopping tablet replica
I20260806 04:33:29.504311 21579 tablet_replica.cc:330] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: stopping tablet replica
I20260806 04:33:29.504570 21579 raft_consensus.cc:2229] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:29.504570 21576 raft_consensus.cc:2229] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:29.504992 21576 raft_consensus.cc:2258] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:29.505028 21579 raft_consensus.cc:2258] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:29.505434 21577 tablet_replica.cc:330] T 53bec4aed1664349bac198e0d2a2394a P 870a8d60724f4bb69296e0a8b8e4469a: stopping tablet replica
I20260806 04:33:29.505609 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20959
I20260806 04:33:29.517455 21576 ts_tablet_manager.cc:1905] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:29.519030 21579 ts_tablet_manager.cc:1905] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:29.520061 21576 ts_tablet_manager.cc:1918] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:29.520174 21576 log.cc:1196] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/53bec4aed1664349bac198e0d2a2394a
I20260806 04:33:29.520473 21576 ts_tablet_manager.cc:1939] T 53bec4aed1664349bac198e0d2a2394a P 58e7dcc5d8d74c5ba78273b9485c1129: Deleting consensus metadata
I20260806 04:33:29.521939 20860 catalog_manager.cc:4841] TS 58e7dcc5d8d74c5ba78273b9485c1129 (127.28.94.2:42945): tablet 53bec4aed1664349bac198e0d2a2394a (table test-table [id=aadcf02f9c674e049723280ee6a38c36]) successfully deleted
I20260806 04:33:29.521997 21579 ts_tablet_manager.cc:1918] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:29.522109 21579 log.cc:1196] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/53bec4aed1664349bac198e0d2a2394a
I20260806 04:33:29.522440 21579 ts_tablet_manager.cc:1939] T 53bec4aed1664349bac198e0d2a2394a P bfa487a3e79b4c699e9434c7c88138f6: Deleting consensus metadata
I20260806 04:33:29.523800 20862 catalog_manager.cc:4841] TS bfa487a3e79b4c699e9434c7c88138f6 (127.28.94.3:46761): tablet 53bec4aed1664349bac198e0d2a2394a (table test-table [id=aadcf02f9c674e049723280ee6a38c36]) successfully deleted
W20260806 04:33:29.529065 20862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:44615 (error 108)
I20260806 04:33:29.529301 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 21138
I20260806 04:33:29.537360 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:29.537011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44615 (local address 127.0.0.1:56586)
0806 04:33:29.537157 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:29.537255 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:33:29.537755 20862 catalog_manager.cc:4584] TS 870a8d60724f4bb69296e0a8b8e4469a (127.28.94.1:44615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53bec4aed1664349bac198e0d2a2394a: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44615: connect: Connection refused (error 111)
I20260806 04:33:29.561749 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 21326
I20260806 04:33:29.577060 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 20836
2026-08-06T04:33:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1271 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:29 dist-test-slave-ncp9 krb5kdc[21585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:29 dist-test-slave-ncp9 krb5kdc[21585](info): set up 2 sockets
Aug 06 04:33:29 dist-test-slave-ncp9 krb5kdc[21585](info): commencing operation
krb5kdc: starting...
W20260806 04:33:30.686375 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:30 dist-test-slave-ncp9 krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:30.819527 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38421
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:30.942297 21600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:30.942651 21600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:30.942716 21600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:30.947038 21600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:30.947124 21600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:30.947153 21600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:30.947176 21600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:30.947209 21600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:30.953136 21600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:30.954420 21600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:30.955585 21600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:30.964721 21608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:30.964777 21600 server_base.cc:1034] running on GCE node
W20260806 04:33:30.964727 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:30.964741 21607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:30.965737 21600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:30.967017 21600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:30.968218 21600 hybrid_clock.cc:648] HybridClock initialized: now 1785990810968190 us; error 44 us; skew 500 ppm
Aug 06 04:33:30 dist-test-slave-ncp9 krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990810, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:30.973991 21600 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:30.975082 21600 webserver.cc:460] Webserver started at http://127.28.94.62:37589/ using document root <none> and password file <none>
I20260806 04:33:30.975718 21600 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:30.975795 21600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:30.976027 21600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:30.978343 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "f0993a55c91d44f8a3c27569383fe89d"
format_stamp: "Formatted at 2026-08-06 04:33:30 on dist-test-slave-ncp9"
server_key: "231927c2b161310aeee1efcc45852e96"
server_key_iv: "6ec4ae223524cc6bd95f72abc335f8f6"
server_key_version: "encryptionkey@0"
I20260806 04:33:30.978971 21600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "f0993a55c91d44f8a3c27569383fe89d"
format_stamp: "Formatted at 2026-08-06 04:33:30 on dist-test-slave-ncp9"
server_key: "231927c2b161310aeee1efcc45852e96"
server_key_iv: "6ec4ae223524cc6bd95f72abc335f8f6"
server_key_version: "encryptionkey@0"
I20260806 04:33:30.983287 21600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:30.986994 21621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:30.988494 21600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:30.988642 21600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "f0993a55c91d44f8a3c27569383fe89d"
format_stamp: "Formatted at 2026-08-06 04:33:30 on dist-test-slave-ncp9"
server_key: "231927c2b161310aeee1efcc45852e96"
server_key_iv: "6ec4ae223524cc6bd95f72abc335f8f6"
server_key_version: "encryptionkey@0"
I20260806 04:33:30.988788 21600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:31.007817 21600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:31.011423 21600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:31.011628 21600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:31.019228 21600 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:38421
I20260806 04:33:31.019302 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:38421 every 8 connection(s)
I20260806 04:33:31.020321 21600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:31.023528 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:31.028069 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 21600
I20260806 04:33:31.028245 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:31.028532 29048 external_mini_cluster.cc:1428] Setting key 09330de89b4b1b20c4cbc5e66faf04bc
I20260806 04:33:31.030236 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: Bootstrap starting.
I20260806 04:33:31.033026 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:31.033888 21692 log.cc:824] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:31.036000 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: No bootstrap required, opened a new log
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:31.040311 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } }
I20260806 04:33:31.040630 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:31.040709 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0993a55c91d44f8a3c27569383fe89d, State: Initialized, Role: FOLLOWER
I20260806 04:33:31.041297 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } }
I20260806 04:33:31.041482 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:31.041559 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:31.041659 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:31.042927 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } }
I20260806 04:33:31.043303 21692 leader_election.cc:302] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0993a55c91d44f8a3c27569383fe89d; no voters:
I20260806 04:33:31.043635 21692 leader_election.cc:288] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:31.043692 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.030146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.28.94.62:38421)
0806 04:33:31.030603 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.030621 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.030663 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:33:31.031385 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.031403 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.031779 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.032126 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:31.033085 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.034191 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.035083 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.035583 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.035748 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.038623 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:31.038631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:31.038650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:31.038686 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:31.040352 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.040937 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.040944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.040952 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.041112 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.041414 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.041416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.041417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.041951 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:31.042129 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.042430 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.042743 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260806 04:33:31.043773 21703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:31.043979 21703 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [term 1 LEADER]: Becoming Leader. State: Replica: f0993a55c91d44f8a3c27569383fe89d, State: Running, Role: LEADER
I20260806 04:33:31.044318 21703 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } }
I20260806 04:33:31.045038 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:31.046886 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0993a55c91d44f8a3c27569383fe89d. Latest consensus state: current_term: 1 leader_uuid: "f0993a55c91d44f8a3c27569383fe89d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } } }
I20260806 04:33:31.047056 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:31.047632 21730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:31.047595 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0993a55c91d44f8a3c27569383fe89d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0993a55c91d44f8a3c27569383fe89d" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 38421 } } }
I20260806 04:33:31.049026 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:31.050887 21730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:31.057313 21730 catalog_manager.cc:1347] Generated new cluster ID: d5ab6936a0554965968ddc8634ed7b76
I20260806 04:33:31.057416 21730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:31.069324 21730 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:31.070545 21730 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:31.081799 21730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: Generated new TSK 0
I20260806 04:33:31.082630 21730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:33:31.119202 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:31.246069 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:31.246397 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:31.246469 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:31.250798 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:31.250900 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:31.250994 21740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:31.256750 21740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:31.258029 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:31.259164 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:31.269543 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:31.269635 21740 server_base.cc:1034] running on GCE node
W20260806 04:33:31.269544 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:31.269544 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:31.270308 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:31.271005 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:31.272182 21740 hybrid_clock.cc:648] HybridClock initialized: now 1785990811272163 us; error 32 us; skew 500 ppm
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:31.275200 21740 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:33:31.276219 21740 webserver.cc:460] Webserver started at http://127.28.94.1:39697/ using document root <none> and password file <none>
I20260806 04:33:31.276827 21740 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:31.276928 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:31.277204 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:31.279469 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "68a8a829764943c3b2113e7a15dfff52"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "ebcbb931c623c6392aa66908f5883954"
server_key_iv: "82a1e0cd1ebc1377427e98a447f37a57"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.280082 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "68a8a829764943c3b2113e7a15dfff52"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "ebcbb931c623c6392aa66908f5883954"
server_key_iv: "82a1e0cd1ebc1377427e98a447f37a57"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.284348 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:33:31.287154 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.288528 21740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:31.288654 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "68a8a829764943c3b2113e7a15dfff52"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "ebcbb931c623c6392aa66908f5883954"
server_key_iv: "82a1e0cd1ebc1377427e98a447f37a57"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.288764 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:31.300731 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:31.304062 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:31.304263 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:31.304879 21740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:31.305866 21740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:31.305917 21740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.305959 21740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:31.305979 21740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.315002 21740 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:36197
I20260806 04:33:31.315026 21930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:36197 every 8 connection(s)
I20260806 04:33:31.316020 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:31.318003 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 21740
I20260806 04:33:31.318146 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:31.318461 29048 external_mini_cluster.cc:1428] Setting key c1e1931bec09ec13008c4322dfa2137e
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:33:31.331672 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.318151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:41911 (local address 127.28.94.62:38421)
0806 04:33:31.318298 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.318302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.319728 (+ 1426us) server_negotiation.cc:407] Connection header received
0806 04:33:31.320828 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.320832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.320910 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.320994 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:31.322333 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.322887 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.324319 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.324564 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.324656 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.327777 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:31.327781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:31.327783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:31.327813 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:31.329265 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.329916 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.329919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.329920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.330013 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.330385 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.330389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.330390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.330602 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:31.330693 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.331343 (+ 650us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.331491 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:33:31.332713 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.318720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38421 (local address 127.28.94.1:41911)
0806 04:33:31.319562 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.319629 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.320586 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.321189 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.321198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.321590 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:31.322158 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.322175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.323024 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.323027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.324182 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.324196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.324747 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.324751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.324996 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.325662 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:31.325685 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:31.327547 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:31.329407 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.329415 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.329439 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.329819 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.330108 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.330112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.330114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.330282 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.330712 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:31.330720 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:31.331082 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.331281 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.331692 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":143,"threads_started":1}
I20260806 04:33:31.334224 21932 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38421
I20260806 04:33:31.334486 21932 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:31.335041 21932 heartbeater.cc:511] Master 127.28.94.62:38421 requested a full tablet report, sending...
I20260806 04:33:31.336952 21646 ts_manager.cc:194] Registered new tserver with Master: 68a8a829764943c3b2113e7a15dfff52 (127.28.94.1:36197)
I20260806 04:33:31.338593 21646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:41911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:33:31.350039 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:31.471289 21939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:31.471609 21939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:31.471676 21939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:31.475901 21939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:31.475996 21939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:31.476082 21939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:31.481769 21939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:31.482991 21939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:31.484112 21939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:31.494750 21944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:31.494760 21947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:31.494765 21945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:31.494982 21939 server_base.cc:1034] running on GCE node
I20260806 04:33:31.495477 21939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:31.496179 21939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:31.497349 21939 hybrid_clock.cc:648] HybridClock initialized: now 1785990811497322 us; error 38 us; skew 500 ppm
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:31.500550 21939 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:33:31.501713 21939 webserver.cc:460] Webserver started at http://127.28.94.2:34875/ using document root <none> and password file <none>
I20260806 04:33:31.502369 21939 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:31.502468 21939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:31.502719 21939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:31.505152 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "254b8176d6984af6ad9c0b80c0adabb2"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "e12976fe2b07ffbaa2cfb671c3fd592d"
server_key_iv: "835a06cb1de1c7e835a2ae54921f277d"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.505822 21939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "254b8176d6984af6ad9c0b80c0adabb2"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "e12976fe2b07ffbaa2cfb671c3fd592d"
server_key_iv: "835a06cb1de1c7e835a2ae54921f277d"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.510191 21939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:33:31.513140 21958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.514635 21939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:33:31.514796 21939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "254b8176d6984af6ad9c0b80c0adabb2"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "e12976fe2b07ffbaa2cfb671c3fd592d"
server_key_iv: "835a06cb1de1c7e835a2ae54921f277d"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.514928 21939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:31.534468 21939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:31.538214 21939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:31.538452 21939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:31.539151 21939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:31.540175 21939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:31.540236 21939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.540303 21939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:31.540334 21939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.551200 21939 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:34253
I20260806 04:33:31.551218 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:34253 every 8 connection(s)
I20260806 04:33:31.552266 21939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:31.558607 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 21939
I20260806 04:33:31.558725 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:31.558993 29048 external_mini_cluster.cc:1428] Setting key cb035cd4012dd59088e59c5be9d77307
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:31.567790 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.554301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:46293 (local address 127.28.94.62:38421)
0806 04:33:31.554448 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.554453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.555578 (+ 1125us) server_negotiation.cc:407] Connection header received
0806 04:33:31.556548 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.556552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.556621 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.556712 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:31.558137 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.558750 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.560151 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.560484 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.560593 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.563698 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:31.563701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:31.563704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:31.563735 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:31.565322 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.565972 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.565976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.565978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.566100 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.566476 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.566479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.566481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.566711 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:31.566811 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.567456 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.567612 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:33:31.568740 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.554659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38421 (local address 127.28.94.2:46293)
0806 04:33:31.555375 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.555422 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.556313 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.556965 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.556975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.557397 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:31.557963 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.557980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.558904 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.558907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.560023 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.560035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.560662 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.560668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.560855 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.561553 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:31.561574 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:31.563449 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:31.565488 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.565511 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.565529 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.565841 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.566196 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.566198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.566199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.566354 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.566831 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:31.566839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:31.567191 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.567387 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.567786 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":117,"threads_started":1}
I20260806 04:33:31.570216 22130 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38421
I20260806 04:33:31.570482 22130 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:33:31.571060 22130 heartbeater.cc:511] Master 127.28.94.62:38421 requested a full tablet report, sending...
I20260806 04:33:31.572327 21646 ts_manager.cc:194] Registered new tserver with Master: 254b8176d6984af6ad9c0b80c0adabb2 (127.28.94.2:34253)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
I20260806 04:33:31.573105 21646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:46293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:33:31.589988 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:31.711890 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:31.712219 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:31.712268 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:31.716527 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:31.716619 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:31.716686 22137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:31.722605 22137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:31.723856 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:31.725026 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:31.735441 22149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:31.735560 22137 server_base.cc:1034] running on GCE node
W20260806 04:33:31.735424 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:31.735424 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:31.736205 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:31.736944 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:31.738121 22137 hybrid_clock.cc:648] HybridClock initialized: now 1785990811738088 us; error 39 us; skew 500 ppm
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:31.741084 22137 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:33:31.742117 22137 webserver.cc:460] Webserver started at http://127.28.94.3:33315/ using document root <none> and password file <none>
I20260806 04:33:31.742738 22137 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:31.742807 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:31.742995 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:31.745388 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "27f2e47b729f4f46a6979e6c8e38407c"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "b5ecba0fe388f60bf11a5332f99e8a68"
server_key_iv: "19f8f9d054177d8b8b9d5fe45c98b1b4"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.746057 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "27f2e47b729f4f46a6979e6c8e38407c"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "b5ecba0fe388f60bf11a5332f99e8a68"
server_key_iv: "19f8f9d054177d8b8b9d5fe45c98b1b4"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.750403 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:31.753453 22159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.754812 22137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:31.754947 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "27f2e47b729f4f46a6979e6c8e38407c"
format_stamp: "Formatted at 2026-08-06 04:33:31 on dist-test-slave-ncp9"
server_key: "b5ecba0fe388f60bf11a5332f99e8a68"
server_key_iv: "19f8f9d054177d8b8b9d5fe45c98b1b4"
server_key_version: "encryptionkey@0"
I20260806 04:33:31.755076 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:31.771708 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:31.774998 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:31.775204 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:31.775878 22137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:31.776847 22137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:31.776933 22137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.777001 22137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:31.777033 22137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:31.786343 22137 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:42927
I20260806 04:33:31.786366 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:42927 every 8 connection(s)
I20260806 04:33:31.787395 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:31.788394 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 22137
I20260806 04:33:31.788539 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:31.788865 29048 external_mini_cluster.cc:1428] Setting key 9fc69025c9a2dc21db307918d3b4a042
Aug 06 04:33:31 dist-test-slave-ncp9 krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990811, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:31.801427 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.789712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:55223 (local address 127.28.94.62:38421)
0806 04:33:31.789894 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.789899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.791092 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 04:33:31.792092 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.792095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.792165 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.792234 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:31.793613 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.794204 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.795351 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.795566 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.795647 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.798215 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:31.798218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:31.798220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:31.798249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:31.799425 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.799914 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.799916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.799917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.799989 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.800258 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.800260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.800261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.800446 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:31.800518 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.801120 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.801250 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:33:31.802527 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.790102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:38421 (local address 127.28.94.3:55223)
0806 04:33:31.790920 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.790983 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.791906 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.792398 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.792404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.792912 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:31.793480 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.793494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.794317 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.794319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.795241 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.795251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.795702 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.795705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.795852 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.796470 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:31.796489 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:31.798030 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:31.799533 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.799539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.799554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.799836 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.800051 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:31.800053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:31.800054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:31.800192 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:31.800517 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:31.800522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:31.800812 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.801030 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.801342 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":130,"threads_started":1}
I20260806 04:33:31.803809 22325 heartbeater.cc:347] Connected to a master server at 127.28.94.62:38421
I20260806 04:33:31.804046 22325 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:31.804562 22325 heartbeater.cc:511] Master 127.28.94.62:38421 requested a full tablet report, sending...
I20260806 04:33:31.805697 21646 ts_manager.cc:194] Registered new tserver with Master: 27f2e47b729f4f46a6979e6c8e38407c (127.28.94.3:42927)
I20260806 04:33:31.806305 21646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:55223
I20260806 04:33:31.815577 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:31.825649 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.817817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.28.94.62:38421)
0806 04:33:31.817940 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.817944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.818097 (+ 153us) server_negotiation.cc:407] Connection header received
0806 04:33:31.818211 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.818213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.818257 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.818326 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:31.819056 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.819578 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.820408 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.820638 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.820717 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.822431 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:31.822434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:31.822437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:31.822467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:31.823835 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.824380 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.824383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.824384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.824468 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:31.824798 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:31.824800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:31.824801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:31.825056 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:31.825167 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.825408 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.825494 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:33:31.828945 21646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:31.832295 21646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:31.847565 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.841582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:42927 (local address 127.0.0.1:59044)
0806 04:33:31.841955 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.842012 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.842156 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.843202 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.843208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.843230 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:31.843419 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.843427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.844753 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.844757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.845978 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.845988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.846896 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.846899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.847099 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.847151 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.847259 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.847402 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":89,"thread_start_us":86,"threads_started":1}
I20260806 04:33:31.847595 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.841749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.28.94.3:42927)
0806 04:33:31.842269 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.842282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.842307 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:33:31.842388 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.842394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.843003 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.843577 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:31.843745 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.844624 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.846119 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.846785 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.846886 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.847056 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.847356 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.847460 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":156,"threads_started":1}
I20260806 04:33:31.848425 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.841794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:34253 (local address 127.0.0.1:60068)
0806 04:33:31.842750 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.842765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.843382 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.843869 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.843872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.843890 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:31.843985 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.843991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.845403 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.845406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.846696 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.846707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.847980 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.847983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.848092 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.848118 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.848205 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.848280 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":35,"thread_start_us":96,"threads_started":1}
I20260806 04:33:31.848513 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.842448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.28.94.2:34253)
0806 04:33:31.842875 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.842886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.843311 (+ 425us) server_negotiation.cc:407] Connection header received
0806 04:33:31.843511 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.843516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.843668 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.843816 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:31.844105 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.845250 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.846832 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.847664 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.847775 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.847917 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.848267 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.848368 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":110,"threads_started":1}
I20260806 04:33:31.849516 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.842376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36197 (local address 127.0.0.1:42186)
0806 04:33:31.843081 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.843094 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.843192 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.843699 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.843702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.843719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:31.843831 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.843838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.845396 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.845400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.847112 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.847122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.849114 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.849118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.849220 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.849245 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.849313 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.849383 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":327,"thread_start_us":102,"threads_started":1}
I20260806 04:33:31.850232 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.842733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.28.94.1:36197)
0806 04:33:31.843077 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.843083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.843166 (+ 83us) server_negotiation.cc:407] Connection header received
0806 04:33:31.843342 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.843348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.843562 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.843710 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:31.843965 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.845258 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.847231 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.848758 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.848875 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.849048 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.849968 (+ 920us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.850087 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260806 04:33:31.851186 22224 tablet_service.cc:1467] Processing CreateTablet for tablet 22799b2c83c343598036dc7141c217da (DEFAULT_TABLE table=test-table [id=5ce0787ca7cb4ab6ab707842b34c4eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:31.851596 22027 tablet_service.cc:1467] Processing CreateTablet for tablet 22799b2c83c343598036dc7141c217da (DEFAULT_TABLE table=test-table [id=5ce0787ca7cb4ab6ab707842b34c4eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:31.852496 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22799b2c83c343598036dc7141c217da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:31.852542 22027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22799b2c83c343598036dc7141c217da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:31.853176 21824 tablet_service.cc:1467] Processing CreateTablet for tablet 22799b2c83c343598036dc7141c217da (DEFAULT_TABLE table=test-table [id=5ce0787ca7cb4ab6ab707842b34c4eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:31.854290 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22799b2c83c343598036dc7141c217da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:31.859402 22354 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Bootstrap starting.
I20260806 04:33:31.861133 22356 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Bootstrap starting.
I20260806 04:33:31.862470 22354 tablet_bootstrap.cc:654] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:31.863415 22354 log.cc:824] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:31.863479 22356 tablet_bootstrap.cc:654] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:31.864172 22358 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: Bootstrap starting.
I20260806 04:33:31.864738 22356 log.cc:824] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:31.865758 22354 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: No bootstrap required, opened a new log
I20260806 04:33:31.865980 22354 ts_tablet_manager.cc:1397] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:33:31.866317 22358 tablet_bootstrap.cc:654] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:31.866950 22356 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: No bootstrap required, opened a new log
I20260806 04:33:31.867017 22358 log.cc:824] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:31.867152 22356 ts_tablet_manager.cc:1397] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 04:33:31.868863 22358 tablet_bootstrap.cc:492] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: No bootstrap required, opened a new log
I20260806 04:33:31.869105 22358 ts_tablet_manager.cc:1397] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260806 04:33:31.870373 22354 raft_consensus.cc:348] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.870704 22354 raft_consensus.cc:374] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:31.870769 22354 raft_consensus.cc:729] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254b8176d6984af6ad9c0b80c0adabb2, State: Initialized, Role: FOLLOWER
I20260806 04:33:31.870960 22356 raft_consensus.cc:348] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.871222 22356 raft_consensus.cc:374] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:31.871272 22356 raft_consensus.cc:729] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f2e47b729f4f46a6979e6c8e38407c, State: Initialized, Role: FOLLOWER
I20260806 04:33:31.871268 22354 consensus_queue.cc:260] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.871810 22356 consensus_queue.cc:260] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.872052 22130 heartbeater.cc:503] Master 127.28.94.62:38421 was elected leader, sending a full tablet report...
I20260806 04:33:31.872437 22354 ts_tablet_manager.cc:1428] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:33:31.872710 22356 ts_tablet_manager.cc:1428] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260806 04:33:31.873067 22325 heartbeater.cc:503] Master 127.28.94.62:38421 was elected leader, sending a full tablet report...
I20260806 04:33:31.873437 22358 raft_consensus.cc:348] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.873657 22358 raft_consensus.cc:374] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:31.873715 22358 raft_consensus.cc:729] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a8a829764943c3b2113e7a15dfff52, State: Initialized, Role: FOLLOWER
I20260806 04:33:31.874140 22358 consensus_queue.cc:260] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.874845 21932 heartbeater.cc:503] Master 127.28.94.62:38421 was elected leader, sending a full tablet report...
I20260806 04:33:31.875147 22358 ts_tablet_manager.cc:1428] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 04:33:31.887817 22367 raft_consensus.cc:482] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:31.888031 22367 raft_consensus.cc:504] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.889269 22367 leader_election.cc:288] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a8a829764943c3b2113e7a15dfff52 (127.28.94.1:36197), 254b8176d6984af6ad9c0b80c0adabb2 (127.28.94.2:34253)
I20260806 04:33:31.893399 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.889639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:56801 (local address 127.28.94.1:36197)
0806 04:33:31.889760 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.889765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.889781 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:33:31.889955 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.889959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.890023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.890092 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:31.890387 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.891149 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.892296 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.892805 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.892927 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.893037 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.893198 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.893277 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:33:31.893399 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.889531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36197 (local address 127.28.94.3:56801)
0806 04:33:31.889682 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.889699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.889810 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.890131 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.890134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.890158 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:31.890283 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.890289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.891267 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.891271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.892167 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.892178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.892950 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.892953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.893066 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.893090 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.893157 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.893242 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260806 04:33:31.893496 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.889786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:54169 (local address 127.28.94.2:34253)
0806 04:33:31.889917 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.889922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.890174 (+ 252us) server_negotiation.cc:407] Connection header received
0806 04:33:31.890250 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.890252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.890295 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.890364 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:31.890599 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.891300 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.892394 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.892857 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.892977 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.893066 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.893299 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.893372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:33:31.893543 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.889531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:34253 (local address 127.28.94.3:54169)
0806 04:33:31.890074 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.890087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:31.890173 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:31.890383 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:31.890386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:31.890404 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:31.890501 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.890505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.891401 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.891405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.892274 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:31.892286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.893006 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.893010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:31.893178 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.893200 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:33:31.893268 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:33:31.893344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":146,"thread_start_us":173,"threads_started":1}
I20260806 04:33:31.894279 22060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22799b2c83c343598036dc7141c217da" candidate_uuid: "27f2e47b729f4f46a6979e6c8e38407c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254b8176d6984af6ad9c0b80c0adabb2" is_pre_election: true
I20260806 04:33:31.894448 21861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22799b2c83c343598036dc7141c217da" candidate_uuid: "27f2e47b729f4f46a6979e6c8e38407c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a8a829764943c3b2113e7a15dfff52" is_pre_election: true
I20260806 04:33:31.894627 22060 raft_consensus.cc:2454] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f2e47b729f4f46a6979e6c8e38407c in term 0.
I20260806 04:33:31.894731 21861 raft_consensus.cc:2454] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f2e47b729f4f46a6979e6c8e38407c in term 0.
I20260806 04:33:31.895159 22161 leader_election.cc:302] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 254b8176d6984af6ad9c0b80c0adabb2, 27f2e47b729f4f46a6979e6c8e38407c, 68a8a829764943c3b2113e7a15dfff52; no voters:
I20260806 04:33:31.895450 22367 raft_consensus.cc:2781] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:31.895548 22367 raft_consensus.cc:482] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:31.895602 22367 raft_consensus.cc:3037] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:31.896881 22367 raft_consensus.cc:504] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.897312 22367 leader_election.cc:288] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [CANDIDATE]: Term 1 election: Requested vote from peers 68a8a829764943c3b2113e7a15dfff52 (127.28.94.1:36197), 254b8176d6984af6ad9c0b80c0adabb2 (127.28.94.2:34253)
I20260806 04:33:31.897712 21861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22799b2c83c343598036dc7141c217da" candidate_uuid: "27f2e47b729f4f46a6979e6c8e38407c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a8a829764943c3b2113e7a15dfff52"
I20260806 04:33:31.897850 21861 raft_consensus.cc:3037] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:31.898953 21861 raft_consensus.cc:2454] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f2e47b729f4f46a6979e6c8e38407c in term 1.
I20260806 04:33:31.899241 22060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22799b2c83c343598036dc7141c217da" candidate_uuid: "27f2e47b729f4f46a6979e6c8e38407c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254b8176d6984af6ad9c0b80c0adabb2"
I20260806 04:33:31.899317 22164 leader_election.cc:302] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f2e47b729f4f46a6979e6c8e38407c, 68a8a829764943c3b2113e7a15dfff52; no voters:
I20260806 04:33:31.899369 22060 raft_consensus.cc:3037] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:31.899518 22367 raft_consensus.cc:2781] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:31.899788 22367 raft_consensus.cc:686] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 1 LEADER]: Becoming Leader. State: Replica: 27f2e47b729f4f46a6979e6c8e38407c, State: Running, Role: LEADER
I20260806 04:33:31.900200 22367 consensus_queue.cc:237] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } }
I20260806 04:33:31.900715 22060 raft_consensus.cc:2454] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f2e47b729f4f46a6979e6c8e38407c in term 1.
I20260806 04:33:31.903965 21639 catalog_manager.cc:5495] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f2e47b729f4f46a6979e6c8e38407c (127.28.94.3). New cstate: current_term: 1 leader_uuid: "27f2e47b729f4f46a6979e6c8e38407c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f2e47b729f4f46a6979e6c8e38407c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 } health_report { overall_health: UNKNOWN } } }
I20260806 04:33:31.918509 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.914796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.28.94.3:42927)
0806 04:33:31.914923 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:31.914928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:31.914943 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:33:31.915101 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:31.915104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:31.915167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:31.915258 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:31.915505 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.916035 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.916951 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:31.917179 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:31.917270 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:31.917619 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:31.918058 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:31.918192 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:31.918296 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:33:31.918337 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:33:31.925237 22060 raft_consensus.cc:1261] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 1 FOLLOWER]: Refusing update from remote peer 27f2e47b729f4f46a6979e6c8e38407c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:31.925316 21861 raft_consensus.cc:1261] T 22799b2c83c343598036dc7141c217da P 68a8a829764943c3b2113e7a15dfff52 [term 1 FOLLOWER]: Refusing update from remote peer 27f2e47b729f4f46a6979e6c8e38407c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:31.926007 22367 consensus_queue.cc:1035] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [LEADER]: Connected to new peer: Peer: permanent_uuid: "254b8176d6984af6ad9c0b80c0adabb2" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:31.926232 22370 consensus_queue.cc:1035] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a8a829764943c3b2113e7a15dfff52" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:31.936018 22375 mvcc.cc:205] Tried to move back new op lower bound from 7315418365637074944 to 7315418365546999808. Current Snapshot: MvccSnapshot[applied={T|T < 7315418365637074944}]
I20260806 04:33:31.937239 22374 mvcc.cc:205] Tried to move back new op lower bound from 7315418365637074944 to 7315418365546999808. Current Snapshot: MvccSnapshot[applied={T|T < 7315418365637074944 or (T in {7315418365637074944})}]
I20260806 04:33:31.937316 22378 mvcc.cc:205] Tried to move back new op lower bound from 7315418365637074944 to 7315418365546999808. Current Snapshot: MvccSnapshot[applied={T|T < 7315418365637074944}]
I20260806 04:33:31.945588 21639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:31.945787 21639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:31.948655 21639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0993a55c91d44f8a3c27569383fe89d: Sending DeleteTablet for 3 replicas of tablet 22799b2c83c343598036dc7141c217da
I20260806 04:33:31.949699 22027 tablet_service.cc:1514] Processing DeleteTablet for tablet 22799b2c83c343598036dc7141c217da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:31 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:60068
I20260806 04:33:31.949699 22224 tablet_service.cc:1514] Processing DeleteTablet for tablet 22799b2c83c343598036dc7141c217da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:31 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:59044
I20260806 04:33:31.950347 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 21740
I20260806 04:33:31.960964 22392 tablet_replica.cc:330] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: stopping tablet replica
I20260806 04:33:31.961555 22392 raft_consensus.cc:2229] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:31.962170 22392 raft_consensus.cc:2258] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:31.965070 22392 ts_tablet_manager.cc:1905] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:31.969234 22393 tablet_replica.cc:330] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: stopping tablet replica
I20260806 04:33:31.969794 22393 raft_consensus.cc:2229] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:31.970314 22393 raft_consensus.cc:2258] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:31.970863 22392 ts_tablet_manager.cc:1918] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:31.971025 22392 log.cc:1196] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/22799b2c83c343598036dc7141c217da
I20260806 04:33:31.971491 22392 ts_tablet_manager.cc:1939] T 22799b2c83c343598036dc7141c217da P 27f2e47b729f4f46a6979e6c8e38407c: Deleting consensus metadata
I20260806 04:33:31.973393 21623 catalog_manager.cc:4841] TS 27f2e47b729f4f46a6979e6c8e38407c (127.28.94.3:42927): tablet 22799b2c83c343598036dc7141c217da (table test-table [id=5ce0787ca7cb4ab6ab707842b34c4eba]) successfully deleted
I20260806 04:33:31.976293 22393 ts_tablet_manager.cc:1905] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:31.979369 22393 ts_tablet_manager.cc:1918] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:31.979507 22393 log.cc:1196] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/22799b2c83c343598036dc7141c217da
I20260806 04:33:31.979871 22393 ts_tablet_manager.cc:1939] T 22799b2c83c343598036dc7141c217da P 254b8176d6984af6ad9c0b80c0adabb2: Deleting consensus metadata
I20260806 04:33:31.981498 21622 catalog_manager.cc:4841] TS 254b8176d6984af6ad9c0b80c0adabb2 (127.28.94.2:34253): tablet 22799b2c83c343598036dc7141c217da (table test-table [id=5ce0787ca7cb4ab6ab707842b34c4eba]) successfully deleted
W20260806 04:33:31.983466 21623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:36197 (error 108)
I20260806 04:33:31.984441 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 21939
I20260806 04:33:31.984725 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:31.984360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:36197 (local address 127.0.0.1:42194)
0806 04:33:31.984528 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:31.984613 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:33:31.992110 21623 catalog_manager.cc:4584] TS 68a8a829764943c3b2113e7a15dfff52 (127.28.94.1:36197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22799b2c83c343598036dc7141c217da: Network error: Client connection negotiation failed: client connection to 127.28.94.1:36197: connect: Connection refused (error 111)
I20260806 04:33:32.001116 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 22137
I20260806 04:33:32.015769 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 21600
2026-08-06T04:33:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2439 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T04:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:32Z Disabled control of system clock
I20260806 04:33:32.057761 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33937
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:32.178017 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:32.178345 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:32.178382 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:32.182667 22402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:32.182755 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:32.182770 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:32.182785 22402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:32.182801 22402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:32.188076 22402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:32.189576 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:32.191128 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:32.200253 22409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.200263 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.200237 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:32.200680 22402 server_base.cc:1034] running on GCE node
I20260806 04:33:32.201347 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:32.202606 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:32.203784 22402 hybrid_clock.cc:648] HybridClock initialized: now 1785990812203765 us; error 30 us; skew 500 ppm
I20260806 04:33:32.205955 22402 webserver.cc:460] Webserver started at http://127.28.94.62:42527/ using document root <none> and password file <none>
I20260806 04:33:32.206619 22402 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:32.206692 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:32.206928 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:32.209281 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "a220d2f3b8b04349a7c13da10754b737"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "851c4c71a2ff144be03e6888130b2454"
server_key_iv: "47b11908f3ab57d0b21a7e1c94d176ac"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.209914 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "a220d2f3b8b04349a7c13da10754b737"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "851c4c71a2ff144be03e6888130b2454"
server_key_iv: "47b11908f3ab57d0b21a7e1c94d176ac"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.213987 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:33:32.216689 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.218139 22402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:33:32.218305 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "a220d2f3b8b04349a7c13da10754b737"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "851c4c71a2ff144be03e6888130b2454"
server_key_iv: "47b11908f3ab57d0b21a7e1c94d176ac"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.218432 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:32.230245 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:32.231114 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:32.231331 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:32.238999 22402 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:33937
I20260806 04:33:32.239012 22495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:33937 every 8 connection(s)
I20260806 04:33:32.240087 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:32.243636 22496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:32.244921 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 22402
I20260806 04:33:32.245047 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:32.245340 29048 external_mini_cluster.cc:1428] Setting key af36665b88d53e61ca1442a239210e7e
I20260806 04:33:32.250363 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: Bootstrap starting.
I20260806 04:33:32.253376 22496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:32.254272 22496 log.cc:824] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:32.255389 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.246954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.28.94.62:33937)
0806 04:33:32.247632 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.247649 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.247690 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:33:32.248371 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.248389 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.248835 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.249195 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:32.249562 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.250545 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.251718 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.252227 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.252351 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.253300 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:32.253306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:32.253322 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:32.253786 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:32.253862 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:32.253886 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:32.253965 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:32.254125 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 04:33:32.254430 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":126,"threads_started":1}
I20260806 04:33:32.256743 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: No bootstrap required, opened a new log
I20260806 04:33:32.260636 22496 raft_consensus.cc:348] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } }
I20260806 04:33:32.260938 22496 raft_consensus.cc:374] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:32.261004 22496 raft_consensus.cc:729] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220d2f3b8b04349a7c13da10754b737, State: Initialized, Role: FOLLOWER
I20260806 04:33:32.261538 22496 consensus_queue.cc:260] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } }
I20260806 04:33:32.261705 22496 raft_consensus.cc:388] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:32.261777 22496 raft_consensus.cc:482] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:32.261895 22496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:32.263005 22496 raft_consensus.cc:504] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } }
I20260806 04:33:32.263357 22496 leader_election.cc:302] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a220d2f3b8b04349a7c13da10754b737; no voters:
I20260806 04:33:32.263698 22496 leader_election.cc:288] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:32.263815 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:32.264045 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [term 1 LEADER]: Becoming Leader. State: Replica: a220d2f3b8b04349a7c13da10754b737, State: Running, Role: LEADER
I20260806 04:33:32.264403 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } }
I20260806 04:33:32.265094 22496 sys_catalog.cc:564] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:32.266402 22503 sys_catalog.cc:455] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a220d2f3b8b04349a7c13da10754b737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } } }
I20260806 04:33:32.266389 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a220d2f3b8b04349a7c13da10754b737. Latest consensus state: current_term: 1 leader_uuid: "a220d2f3b8b04349a7c13da10754b737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a220d2f3b8b04349a7c13da10754b737" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33937 } } }
I20260806 04:33:32.266557 22503 sys_catalog.cc:458] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:32.266557 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:32.267050 22528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:32.270519 22528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:32.276445 22528 catalog_manager.cc:1347] Generated new cluster ID: 2322e337a09f4f849c30722b235c88f8
I20260806 04:33:32.276539 22528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:32.295768 22528 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:32.296777 22528 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:32.324281 22528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: Generated new TSK 0
I20260806 04:33:32.325161 22528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:32.335883 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--builtin_ntp_servers=127.28.94.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:32.458626 22540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:32.458938 22540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:32.458995 22540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:32.463274 22540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:32.463352 22540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:32.463431 22540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:32.468662 22540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:32.469944 22540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:32.471453 22540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:32.481943 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.481943 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.481945 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:32.482378 22540 server_base.cc:1034] running on GCE node
I20260806 04:33:32.482906 22540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:32.483640 22540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:32.484812 22540 hybrid_clock.cc:648] HybridClock initialized: now 1785990812484794 us; error 39 us; skew 500 ppm
I20260806 04:33:32.486896 22540 webserver.cc:460] Webserver started at http://127.28.94.1:39431/ using document root <none> and password file <none>
I20260806 04:33:32.487530 22540 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:32.487619 22540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:32.487843 22540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:32.490149 22540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "a113484bc84a42f9861ead7d0874af4d"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "fefc78b7ecfbaedf997b35c9430ad0c7"
server_key_iv: "42de014894e9c51945a4d200aedd10d7"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.490757 22540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "a113484bc84a42f9861ead7d0874af4d"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "fefc78b7ecfbaedf997b35c9430ad0c7"
server_key_iv: "42de014894e9c51945a4d200aedd10d7"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.494971 22540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:32.497804 22555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.499145 22540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:32.499282 22540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "a113484bc84a42f9861ead7d0874af4d"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "fefc78b7ecfbaedf997b35c9430ad0c7"
server_key_iv: "42de014894e9c51945a4d200aedd10d7"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.499402 22540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:32.513854 22540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:32.514652 22540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:32.514851 22540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:32.515504 22540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:32.516607 22540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:32.516661 22540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.516722 22540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:32.516749 22540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.526232 22540 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:44923
I20260806 04:33:32.526257 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:44923 every 8 connection(s)
I20260806 04:33:32.527295 22540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:32.534189 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 22540
I20260806 04:33:32.534319 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:32.534605 29048 external_mini_cluster.cc:1428] Setting key d4d6529dc6d184f5b3511fe36920faed
I20260806 04:33:32.536959 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--builtin_ntp_servers=127.28.94.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:32.537603 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.529383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:34949 (local address 127.28.94.62:33937)
0806 04:33:32.529538 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.529542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.530726 (+ 1184us) server_negotiation.cc:407] Connection header received
0806 04:33:32.532221 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.532224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.532283 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.532362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:32.533385 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.533970 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.535292 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.535557 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.535647 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.537025 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:32.537030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:32.537033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:32.537051 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:32.537076 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:32.537086 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:32.537159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:32.537372 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 04:33:32.537448 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:33:32.538544 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.529835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33937 (local address 127.28.94.1:34949)
0806 04:33:32.530552 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.530615 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.531990 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.532534 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.532542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.532650 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:32.533253 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.533268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.534081 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.534083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.535168 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.535182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.535734 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.535739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.535953 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.536690 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:32.536691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:32.536740 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:32.536819 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:32.537211 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:32.537223 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:32.537230 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:33:32.537352 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 04:33:32.537628 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
I20260806 04:33:32.539872 22717 heartbeater.cc:347] Connected to a master server at 127.28.94.62:33937
I20260806 04:33:32.540194 22717 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:32.540824 22717 heartbeater.cc:511] Master 127.28.94.62:33937 requested a full tablet report, sending...
I20260806 04:33:32.542645 22444 ts_manager.cc:194] Registered new tserver with Master: a113484bc84a42f9861ead7d0874af4d (127.28.94.1:44923)
I20260806 04:33:32.543900 22444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.1:34949
W20260806 04:33:32.657903 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:32.658224 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:32.658281 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:32.662528 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:32.662621 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:32.662699 22724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:32.667902 22724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:32.669196 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:32.670737 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:32.681354 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.681358 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:32.681533 22724 server_base.cc:1034] running on GCE node
W20260806 04:33:32.681354 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:32.682181 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:32.682950 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:32.684121 22724 hybrid_clock.cc:648] HybridClock initialized: now 1785990812684098 us; error 35 us; skew 500 ppm
I20260806 04:33:32.686162 22724 webserver.cc:460] Webserver started at http://127.28.94.2:35825/ using document root <none> and password file <none>
I20260806 04:33:32.686810 22724 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:32.686889 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:32.687127 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:32.689512 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "5b0ed7c6ac5b4ecfa9541b769308af74"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "f932ddfbbf029690ca7f20595e50515a"
server_key_iv: "c392bafe49d84312200ce01624f036cc"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.690156 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "5b0ed7c6ac5b4ecfa9541b769308af74"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "f932ddfbbf029690ca7f20595e50515a"
server_key_iv: "c392bafe49d84312200ce01624f036cc"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.694499 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:32.697505 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.699332 22724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:32.699503 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "5b0ed7c6ac5b4ecfa9541b769308af74"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "f932ddfbbf029690ca7f20595e50515a"
server_key_iv: "c392bafe49d84312200ce01624f036cc"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.699649 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:32.720187 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:32.721051 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:32.721266 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:32.721961 22724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:32.723080 22724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:32.723131 22724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.723201 22724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:32.723227 22724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.732612 22724 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:38689
I20260806 04:33:32.732645 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:38689 every 8 connection(s)
I20260806 04:33:32.733800 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:32.735297 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 22724
I20260806 04:33:32.735431 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:32.735754 29048 external_mini_cluster.cc:1428] Setting key d318f7d19528bcbae0550a73747a7b70
I20260806 04:33:32.737895 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--builtin_ntp_servers=127.28.94.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:32.743934 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.735886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:55663 (local address 127.28.94.62:33937)
0806 04:33:32.736062 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.736067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.737403 (+ 1336us) server_negotiation.cc:407] Connection header received
0806 04:33:32.738890 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.738894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.738956 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.739044 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:32.740049 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.740587 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.741868 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.742099 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.742210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.743415 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:32.743419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:32.743421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:32.743437 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:32.743459 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:32.743467 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:32.743528 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:32.743690 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 04:33:32.743766 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:33:32.744946 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.736356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33937 (local address 127.28.94.2:55663)
0806 04:33:32.737238 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.737300 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.738649 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.739232 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.739240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.739335 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:32.739883 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.739897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.740736 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.740740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.741730 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.741742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.742289 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.742294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.742468 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.743107 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:32.743108 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:32.743163 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:32.743244 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:32.743540 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:32.743551 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:32.743559 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:33:32.743680 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 04:33:32.744010 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":159,"threads_started":1}
I20260806 04:33:32.746404 22905 heartbeater.cc:347] Connected to a master server at 127.28.94.62:33937
I20260806 04:33:32.746677 22905 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:32.747279 22905 heartbeater.cc:511] Master 127.28.94.62:33937 requested a full tablet report, sending...
I20260806 04:33:32.748594 22444 ts_manager.cc:194] Registered new tserver with Master: 5b0ed7c6ac5b4ecfa9541b769308af74 (127.28.94.2:38689)
I20260806 04:33:32.749477 22444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.2:55663
W20260806 04:33:32.862344 22908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:32.862685 22908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:32.862725 22908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:32.867058 22908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:32.867139 22908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:32.867200 22908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:32.872278 22908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:32.873596 22908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:32.875106 22908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:32.886173 22918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.886166 22914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:32.886174 22920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:32.886427 22908 server_base.cc:1034] running on GCE node
I20260806 04:33:32.887104 22908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:32.887868 22908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:32.889030 22908 hybrid_clock.cc:648] HybridClock initialized: now 1785990812889008 us; error 38 us; skew 500 ppm
I20260806 04:33:32.891137 22908 webserver.cc:460] Webserver started at http://127.28.94.3:43967/ using document root <none> and password file <none>
I20260806 04:33:32.891772 22908 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:32.891849 22908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:32.892084 22908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:32.894408 22908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "3b44dbf7609e4dc4961b28d33154ac9a"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "8be60a8f59c9c95f3226990eda4bcf45"
server_key_iv: "bb9ad420ff02e1ba0ccc70decb590606"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.895066 22908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "3b44dbf7609e4dc4961b28d33154ac9a"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "8be60a8f59c9c95f3226990eda4bcf45"
server_key_iv: "bb9ad420ff02e1ba0ccc70decb590606"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.899333 22908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:32.902226 22928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.903854 22908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:33:32.903998 22908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "3b44dbf7609e4dc4961b28d33154ac9a"
format_stamp: "Formatted at 2026-08-06 04:33:32 on dist-test-slave-ncp9"
server_key: "8be60a8f59c9c95f3226990eda4bcf45"
server_key_iv: "bb9ad420ff02e1ba0ccc70decb590606"
server_key_version: "encryptionkey@0"
I20260806 04:33:32.904119 22908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:32.934755 22908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:32.935541 22908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:32.935750 22908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:32.936411 22908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:32.937517 22908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:32.937572 22908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.937641 22908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:32.937666 22908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:32.946811 22908 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:42037
I20260806 04:33:32.946837 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:42037 every 8 connection(s)
I20260806 04:33:32.947857 22908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:32.956993 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 22908
I20260806 04:33:32.957202 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:32.957803 29048 external_mini_cluster.cc:1428] Setting key a1cc20a573e3e375180cb324f061e56f
I20260806 04:33:32.959631 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.949931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:45717 (local address 127.28.94.62:33937)
0806 04:33:32.950076 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.950080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.951535 (+ 1455us) server_negotiation.cc:407] Connection header received
0806 04:33:32.953017 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.953021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.953076 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.953158 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:32.954146 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.954667 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.955894 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.956159 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.956285 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.958832 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:32.958836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:32.958839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:32.958860 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:32.958895 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:32.958908 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:32.958996 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:32.959355 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 04:33:32.959467 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:33:32.960796 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.950375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33937 (local address 127.28.94.3:45717)
0806 04:33:32.951355 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.951404 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.952743 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.953334 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.953342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.953434 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:32.953991 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.954006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.954786 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.954789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.955735 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.955747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.956582 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.956591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:32.957132 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.958187 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:32.958190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:32.958405 (+ 215us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:32.958551 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:32.959033 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:32.959045 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:32.959053 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:33:32.959257 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 04:33:32.959831 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":147,"threads_started":1}
I20260806 04:33:32.962293 23090 heartbeater.cc:347] Connected to a master server at 127.28.94.62:33937
I20260806 04:33:32.962589 23090 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:32.963222 23090 heartbeater.cc:511] Master 127.28.94.62:33937 requested a full tablet report, sending...
I20260806 04:33:32.964499 22444 ts_manager.cc:194] Registered new tserver with Master: 3b44dbf7609e4dc4961b28d33154ac9a (127.28.94.3:42037)
I20260806 04:33:32.965241 22444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.3:45717
I20260806 04:33:32.973532 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:32.979382 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.975725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.28.94.62:33937)
0806 04:33:32.975857 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.975861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.976005 (+ 144us) server_negotiation.cc:407] Connection header received
0806 04:33:32.976113 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.976115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.976159 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.976243 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:32.976491 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.977078 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.977857 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.978066 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.978135 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:32.978956 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:32.978959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:32.978961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:32.978978 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:32.978998 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:32.979007 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:32.979109 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:32.979195 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 04:33:32.979239 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:33:32.982584 22444 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:32.985998 22444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:33.002557 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.996748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.28.94.3:42037)
0806 04:33:32.997220 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.997235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.997262 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:33:32.997350 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.997356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.997589 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.997768 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:32.998172 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999127 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.001040 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.001871 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.002003 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.002228 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.002282 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.002379 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":175,"threads_started":1}
I20260806 04:33:33.002559 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.996329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.28.94.1:44923)
0806 04:33:32.996858 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.996871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.997768 (+ 897us) server_negotiation.cc:407] Connection header received
0806 04:33:32.998006 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.998013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.998221 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.998338 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:32.998745 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999658 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.000922 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.001563 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.001632 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.001809 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.002276 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.002392 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":153,"threads_started":1}
I20260806 04:33:33.002556 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.996160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:42037 (local address 127.0.0.1:36494)
0806 04:33:32.997028 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.997042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.997133 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.997924 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.997927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.997943 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:32.998048 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.998055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999240 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.999244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.000863 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.000875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.002009 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.002012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.002090 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.002107 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 04:33:33.002178 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:33:33.002352 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":136,"thread_start_us":123,"threads_started":1}
I20260806 04:33:33.002555 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.995774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:38689 (local address 127.0.0.1:37658)
0806 04:33:32.996256 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.996318 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.996468 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.997806 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.997812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.997851 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:32.998011 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.998020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999276 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.999279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.000863 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.000875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.002019 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.002022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.002156 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.002180 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:33:33.002272 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:33:33.002356 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":179,"spinlock_wait_cycles":4352,"thread_start_us":91,"threads_started":1}
I20260806 04:33:33.003185 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.996740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44923 (local address 127.0.0.1:56344)
0806 04:33:32.997103 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:32.997686 (+ 583us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:32.997857 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:32.998520 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:32.998523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:32.998541 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:32.998621 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:32.998626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999775 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:32.999779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.000754 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.000763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.001892 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.001894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.001993 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.002063 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 04:33:33.002995 (+ 932us) client_negotiation.cc:240] Negotiation successful
0806 04:33:33.003073 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4,"spinlock_wait_cycles":8320,"thread_start_us":118,"threads_started":1}
I20260806 04:33:33.004345 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:32.996528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.28.94.2:38689)
0806 04:33:32.997329 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:32.997338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:32.997357 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:33:32.997421 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:32.997425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:32.997617 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:32.998164 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:32.998286 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:32.999183 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.001041 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.001871 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.003841 (+ 1970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.004021 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.004077 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.004183 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":128,"threads_started":1}
I20260806 04:33:33.006136 22623 tablet_service.cc:1467] Processing CreateTablet for tablet 3428e7c45d9d451c957c6243729a72b8 (DEFAULT_TABLE table=test-table [id=4ffb2f14e8de4fcba4ce84ca389ec404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:33.006194 22994 tablet_service.cc:1467] Processing CreateTablet for tablet 3428e7c45d9d451c957c6243729a72b8 (DEFAULT_TABLE table=test-table [id=4ffb2f14e8de4fcba4ce84ca389ec404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:33.007200 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428e7c45d9d451c957c6243729a72b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:33.007293 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428e7c45d9d451c957c6243729a72b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:33.007325 22810 tablet_service.cc:1467] Processing CreateTablet for tablet 3428e7c45d9d451c957c6243729a72b8 (DEFAULT_TABLE table=test-table [id=4ffb2f14e8de4fcba4ce84ca389ec404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:33.008409 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3428e7c45d9d451c957c6243729a72b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:33.014940 23116 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Bootstrap starting.
I20260806 04:33:33.015626 23118 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Bootstrap starting.
I20260806 04:33:33.015938 23119 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Bootstrap starting.
I20260806 04:33:33.017645 23116 tablet_bootstrap.cc:654] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:33.017760 23119 tablet_bootstrap.cc:654] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:33.017889 23118 tablet_bootstrap.cc:654] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:33.018518 23119 log.cc:824] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:33.018602 23118 log.cc:824] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:33.018602 23116 log.cc:824] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:33.020716 23116 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: No bootstrap required, opened a new log
I20260806 04:33:33.020721 23119 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: No bootstrap required, opened a new log
I20260806 04:33:33.020988 23116 ts_tablet_manager.cc:1397] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:33:33.020987 23119 ts_tablet_manager.cc:1397] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:33:33.022588 23118 tablet_bootstrap.cc:492] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: No bootstrap required, opened a new log
I20260806 04:33:33.022815 23118 ts_tablet_manager.cc:1397] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:33:33.024928 23116 raft_consensus.cc:348] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.025254 23116 raft_consensus.cc:374] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:33.025311 23116 raft_consensus.cc:729] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a113484bc84a42f9861ead7d0874af4d, State: Initialized, Role: FOLLOWER
I20260806 04:33:33.025451 23119 raft_consensus.cc:348] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.025749 23119 raft_consensus.cc:374] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:33.025811 23119 raft_consensus.cc:729] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0ed7c6ac5b4ecfa9541b769308af74, State: Initialized, Role: FOLLOWER
I20260806 04:33:33.025803 23116 consensus_queue.cc:260] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.026253 23119 consensus_queue.cc:260] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.026715 22717 heartbeater.cc:503] Master 127.28.94.62:33937 was elected leader, sending a full tablet report...
I20260806 04:33:33.027012 23116 ts_tablet_manager.cc:1428] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:33:33.027252 22905 heartbeater.cc:503] Master 127.28.94.62:33937 was elected leader, sending a full tablet report...
I20260806 04:33:33.027288 23119 ts_tablet_manager.cc:1428] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260806 04:33:33.028774 22719 tablet.cc:2369] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:33.028630 23118 raft_consensus.cc:348] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.028963 23118 raft_consensus.cc:374] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:33.029016 23118 raft_consensus.cc:729] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b44dbf7609e4dc4961b28d33154ac9a, State: Initialized, Role: FOLLOWER
I20260806 04:33:33.029552 23118 consensus_queue.cc:260] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.030720 23118 ts_tablet_manager.cc:1428] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 04:33:33.031558 23090 heartbeater.cc:503] Master 127.28.94.62:33937 was elected leader, sending a full tablet report...
I20260806 04:33:33.179510 23128 raft_consensus.cc:482] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:33.179765 23128 raft_consensus.cc:504] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.181064 23128 leader_election.cc:288] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b0ed7c6ac5b4ecfa9541b769308af74 (127.28.94.2:38689), a113484bc84a42f9861ead7d0874af4d (127.28.94.1:44923)
I20260806 04:33:33.184808 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.181464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:38467 (local address 127.28.94.2:38689)
0806 04:33:33.181581 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:33.181586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:33.181670 (+ 84us) server_negotiation.cc:407] Connection header received
0806 04:33:33.181842 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:33.181845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:33.181894 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:33.181961 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:33.182298 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.182823 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.183873 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.184257 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.184351 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.184419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.184598 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.184686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:33:33.184808 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.181343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:38689 (local address 127.28.94.3:38467)
0806 04:33:33.181580 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:33.181596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:33.181707 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:33.181997 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:33.182001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:33.182019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:33.182150 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.182159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.182939 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.182943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.183739 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.183749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.184381 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.184385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.184482 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.184502 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:33:33.184568 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:33:33.184654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 04:33:33.184976 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.181459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:52299 (local address 127.28.94.1:44923)
0806 04:33:33.181606 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:33.181621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:33.181905 (+ 284us) server_negotiation.cc:407] Connection header received
0806 04:33:33.182010 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:33.182013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:33.182057 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:33.182120 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:33.182506 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.182988 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.183991 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.184400 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.184492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.184555 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.184751 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.184822 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:33:33.185009 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.181343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44923 (local address 127.28.94.3:52299)
0806 04:33:33.181810 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:33.181822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:33.181918 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:33.182140 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:33.182143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:33.182160 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:33.182386 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.182394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.183098 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.183102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.183861 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:33.183870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.184531 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.184535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:33.184626 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.184653 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:33:33.184717 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:33:33.184788 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":43,"thread_start_us":149,"threads_started":1}
I20260806 04:33:33.185737 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428e7c45d9d451c957c6243729a72b8" candidate_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" is_pre_election: true
I20260806 04:33:33.185734 22650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428e7c45d9d451c957c6243729a72b8" candidate_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a113484bc84a42f9861ead7d0874af4d" is_pre_election: true
I20260806 04:33:33.186048 22840 raft_consensus.cc:2454] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b44dbf7609e4dc4961b28d33154ac9a in term 0.
I20260806 04:33:33.186050 22650 raft_consensus.cc:2454] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b44dbf7609e4dc4961b28d33154ac9a in term 0.
I20260806 04:33:33.186565 22935 leader_election.cc:302] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b44dbf7609e4dc4961b28d33154ac9a, 5b0ed7c6ac5b4ecfa9541b769308af74; no voters:
I20260806 04:33:33.186884 23128 raft_consensus.cc:2781] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:33.186967 23128 raft_consensus.cc:482] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:33.187019 23128 raft_consensus.cc:3037] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:33.188088 23128 raft_consensus.cc:504] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.188491 23128 leader_election.cc:288] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [CANDIDATE]: Term 1 election: Requested vote from peers 5b0ed7c6ac5b4ecfa9541b769308af74 (127.28.94.2:38689), a113484bc84a42f9861ead7d0874af4d (127.28.94.1:44923)
I20260806 04:33:33.188849 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428e7c45d9d451c957c6243729a72b8" candidate_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74"
I20260806 04:33:33.188850 22650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3428e7c45d9d451c957c6243729a72b8" candidate_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a113484bc84a42f9861ead7d0874af4d"
I20260806 04:33:33.189013 22840 raft_consensus.cc:3037] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:33.189013 22650 raft_consensus.cc:3037] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:33.190120 22840 raft_consensus.cc:2454] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b44dbf7609e4dc4961b28d33154ac9a in term 1.
I20260806 04:33:33.190128 22650 raft_consensus.cc:2454] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b44dbf7609e4dc4961b28d33154ac9a in term 1.
I20260806 04:33:33.190471 22935 leader_election.cc:302] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b44dbf7609e4dc4961b28d33154ac9a, 5b0ed7c6ac5b4ecfa9541b769308af74, a113484bc84a42f9861ead7d0874af4d; no voters:
I20260806 04:33:33.190682 23128 raft_consensus.cc:2781] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:33.190927 23128 raft_consensus.cc:686] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 1 LEADER]: Becoming Leader. State: Replica: 3b44dbf7609e4dc4961b28d33154ac9a, State: Running, Role: LEADER
I20260806 04:33:33.191282 23128 consensus_queue.cc:237] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } }
I20260806 04:33:33.194846 22441 catalog_manager.cc:5495] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b44dbf7609e4dc4961b28d33154ac9a (127.28.94.3). New cstate: current_term: 1 leader_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b44dbf7609e4dc4961b28d33154ac9a" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 42037 } health_report { overall_health: HEALTHY } } }
W20260806 04:33:33.198901 23092 tablet.cc:2369] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:33.235011 22906 tablet.cc:2369] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:33.244777 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.240497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.28.94.3:42037)
0806 04:33:33.240666 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:33.240671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:33.240689 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:33:33.240792 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:33.240796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:33.240880 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:33.241027 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:33.241239 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.241998 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.242875 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:33.243220 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:33.243323 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:33.243695 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:33.244261 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:33.244426 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:33.244561 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 04:33:33.244605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:33:33.251942 22650 raft_consensus.cc:1261] T 3428e7c45d9d451c957c6243729a72b8 P a113484bc84a42f9861ead7d0874af4d [term 1 FOLLOWER]: Refusing update from remote peer 3b44dbf7609e4dc4961b28d33154ac9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:33.251942 22840 raft_consensus.cc:1261] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 1 FOLLOWER]: Refusing update from remote peer 3b44dbf7609e4dc4961b28d33154ac9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:33.252805 23128 consensus_queue.cc:1035] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a113484bc84a42f9861ead7d0874af4d" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:33.253075 23132 consensus_queue.cc:1035] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0ed7c6ac5b4ecfa9541b769308af74" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:33.263127 23138 mvcc.cc:205] Tried to move back new op lower bound from 7315418371070914560 to 7315418370834223104. Current Snapshot: MvccSnapshot[applied={T|T < 7315418371070914560}]
I20260806 04:33:33.271278 22441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:33.271467 22441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:33.274322 22441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a220d2f3b8b04349a7c13da10754b737: Sending DeleteTablet for 3 replicas of tablet 3428e7c45d9d451c957c6243729a72b8
I20260806 04:33:33.275279 22623 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428e7c45d9d451c957c6243729a72b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:33 UTC) from {username='slave'} at 127.0.0.1:56344
I20260806 04:33:33.275278 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428e7c45d9d451c957c6243729a72b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:33 UTC) from {username='slave'} at 127.0.0.1:36494
I20260806 04:33:33.275275 22810 tablet_service.cc:1514] Processing DeleteTablet for tablet 3428e7c45d9d451c957c6243729a72b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:33 UTC) from {username='slave'} at 127.0.0.1:37658
I20260806 04:33:33.275867 23151 tablet_replica.cc:330] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: stopping tablet replica
I20260806 04:33:33.276249 23151 raft_consensus.cc:2229] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:33.276751 23151 raft_consensus.cc:2258] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:33.276794 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 22540
I20260806 04:33:33.277118 23150 tablet_replica.cc:330] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: stopping tablet replica
I20260806 04:33:33.277483 23150 raft_consensus.cc:2229] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:33.277801 23150 raft_consensus.cc:2258] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:33.285384 23151 ts_tablet_manager.cc:1905] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:33.288166 23151 ts_tablet_manager.cc:1918] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:33.288278 23151 log.cc:1196] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/3428e7c45d9d451c957c6243729a72b8
I20260806 04:33:33.288544 23151 ts_tablet_manager.cc:1939] T 3428e7c45d9d451c957c6243729a72b8 P 3b44dbf7609e4dc4961b28d33154ac9a: Deleting consensus metadata
I20260806 04:33:33.289909 22422 catalog_manager.cc:4841] TS 3b44dbf7609e4dc4961b28d33154ac9a (127.28.94.3:42037): tablet 3428e7c45d9d451c957c6243729a72b8 (table test-table [id=4ffb2f14e8de4fcba4ce84ca389ec404]) successfully deleted
I20260806 04:33:33.293879 23150 ts_tablet_manager.cc:1905] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:33.296976 23150 ts_tablet_manager.cc:1918] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:33.297107 23150 log.cc:1196] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/3428e7c45d9d451c957c6243729a72b8
I20260806 04:33:33.297420 23150 ts_tablet_manager.cc:1939] T 3428e7c45d9d451c957c6243729a72b8 P 5b0ed7c6ac5b4ecfa9541b769308af74: Deleting consensus metadata
I20260806 04:33:33.300020 22425 catalog_manager.cc:4841] TS 5b0ed7c6ac5b4ecfa9541b769308af74 (127.28.94.2:38689): tablet 3428e7c45d9d451c957c6243729a72b8 (table test-table [id=4ffb2f14e8de4fcba4ce84ca389ec404]) successfully deleted
W20260806 04:33:33.309535 22423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:44923 (error 108)
I20260806 04:33:33.309626 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 22724
I20260806 04:33:33.311144 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:33.310794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:44923 (local address 127.0.0.1:56358)
0806 04:33:33.310990 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:33.311051 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 04:33:33.311311 22423 catalog_manager.cc:4584] TS a113484bc84a42f9861ead7d0874af4d (127.28.94.1:44923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3428e7c45d9d451c957c6243729a72b8: Network error: Client connection negotiation failed: client connection to 127.28.94.1:44923: connect: Connection refused (error 111)
I20260806 04:33:33.324018 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 22908
I20260806 04:33:33.338948 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 22402
2026-08-06T04:33:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:33 dist-test-slave-ncp9 krb5kdc[23158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:33 dist-test-slave-ncp9 krb5kdc[23158](info): set up 2 sockets
Aug 06 04:33:33 dist-test-slave-ncp9 krb5kdc[23158](info): commencing operation
krb5kdc: starting...
W20260806 04:33:34.457821 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:34 dist-test-slave-ncp9 krb5kdc[23158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:34.591914 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36243
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:34.712268 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:34.712594 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:34.712646 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:34.716924 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:34.717023 23177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:34.717043 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:34.717059 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:34.717074 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:34.722862 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:36243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:34.724136 23177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:34.725276 23177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:34.734463 23188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:34.734489 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:34.734467 23183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:34.734638 23177 server_base.cc:1034] running on GCE node
I20260806 04:33:34.735553 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:34.736953 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:34.738130 23177 hybrid_clock.cc:648] HybridClock initialized: now 1785990814738105 us; error 47 us; skew 500 ppm
Aug 06 04:33:34 dist-test-slave-ncp9 krb5kdc[23158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990814, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:34.741547 23177 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:34.742604 23177 webserver.cc:460] Webserver started at http://127.28.94.62:41531/ using document root <none> and password file <none>
I20260806 04:33:34.743208 23177 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:34.743275 23177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:34.743475 23177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:34.745844 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "f964aebaaead4c6bbd29e52e06da6c25"
format_stamp: "Formatted at 2026-08-06 04:33:34 on dist-test-slave-ncp9"
server_key: "89ec21d2a9150dd3cfb8abf62cbacf57"
server_key_iv: "cf643f963326526d7f197cd0ed5198e0"
server_key_version: "encryptionkey@0"
I20260806 04:33:34.746423 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "f964aebaaead4c6bbd29e52e06da6c25"
format_stamp: "Formatted at 2026-08-06 04:33:34 on dist-test-slave-ncp9"
server_key: "89ec21d2a9150dd3cfb8abf62cbacf57"
server_key_iv: "cf643f963326526d7f197cd0ed5198e0"
server_key_version: "encryptionkey@0"
I20260806 04:33:34.750627 23177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:34.753377 23195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:34.754848 23177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:34.754977 23177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "f964aebaaead4c6bbd29e52e06da6c25"
format_stamp: "Formatted at 2026-08-06 04:33:34 on dist-test-slave-ncp9"
server_key: "89ec21d2a9150dd3cfb8abf62cbacf57"
server_key_iv: "cf643f963326526d7f197cd0ed5198e0"
server_key_version: "encryptionkey@0"
I20260806 04:33:34.755089 23177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:34.773751 23177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:34.777254 23177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:34.777437 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:34.785141 23177 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:36243
I20260806 04:33:34.785171 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:36243 every 8 connection(s)
I20260806 04:33:34.786226 23177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:34.789450 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:34.790280 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 23177
I20260806 04:33:34.790410 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:34.790705 29048 external_mini_cluster.cc:1428] Setting key a3c60bf8833f27f9e59281dc0690e57d
I20260806 04:33:34.796988 23275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: Bootstrap starting.
I20260806 04:33:34.799804 23275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: Neither blocks nor log segments found. Creating new log.
Aug 06 04:33:34 dist-test-slave-ncp9 krb5kdc[23158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:34.800832 23275 log.cc:824] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:34.803381 23275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: No bootstrap required, opened a new log
I20260806 04:33:34.806196 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:34.792241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.28.94.62:36243)
0806 04:33:34.792735 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:34.792756 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:34.792806 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:33:34.793703 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:34.793730 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:34.794217 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:34.794613 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:34.795299 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:34.796780 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:34.797726 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:34.798476 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:34.798660 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:34.801396 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:34.801404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:34.801425 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:34.801466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:34.803182 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:34.803748 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:34.803753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:34.803758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:34.803879 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:34.804151 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:34.804161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:34.804162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:34.804574 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:34.804722 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:34.805045 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 04:33:34.805393 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":120,"threads_started":1}
I20260806 04:33:34.807614 23275 raft_consensus.cc:348] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } }
I20260806 04:33:34.807869 23275 raft_consensus.cc:374] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:34.807941 23275 raft_consensus.cc:729] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f964aebaaead4c6bbd29e52e06da6c25, State: Initialized, Role: FOLLOWER
I20260806 04:33:34.808424 23275 consensus_queue.cc:260] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } }
I20260806 04:33:34.808588 23275 raft_consensus.cc:388] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:34.808655 23275 raft_consensus.cc:482] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:34.808763 23275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:34.809836 23275 raft_consensus.cc:504] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } }
I20260806 04:33:34.810252 23275 leader_election.cc:302] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f964aebaaead4c6bbd29e52e06da6c25; no voters:
I20260806 04:33:34.810611 23275 leader_election.cc:288] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:34.810726 23280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:34.810953 23280 raft_consensus.cc:686] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [term 1 LEADER]: Becoming Leader. State: Replica: f964aebaaead4c6bbd29e52e06da6c25, State: Running, Role: LEADER
I20260806 04:33:34.811264 23280 consensus_queue.cc:237] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } }
I20260806 04:33:34.811944 23275 sys_catalog.cc:564] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:34.813040 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f964aebaaead4c6bbd29e52e06da6c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } } }
I20260806 04:33:34.813187 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:34.813486 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f964aebaaead4c6bbd29e52e06da6c25. Latest consensus state: current_term: 1 leader_uuid: "f964aebaaead4c6bbd29e52e06da6c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f964aebaaead4c6bbd29e52e06da6c25" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 36243 } } }
I20260806 04:33:34.813580 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:34.813889 23291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:34.817369 23291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:34.823755 23291 catalog_manager.cc:1347] Generated new cluster ID: a6c9549d4c8f44beb0a1e0c6efc0be7d
I20260806 04:33:34.823843 23291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:34.842484 23291 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:34.843906 23291 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:34.855623 23291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: Generated new TSK 0
I20260806 04:33:34.856442 23291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.94.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
Entry for principal kudu/127.28.94.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab.
I20260806 04:33:34.898205 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:35.023456 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:35.023801 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:35.023854 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:35.028146 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:35.028229 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:35.028292 23312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:35.034032 23312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.1.keytab
--principal=kudu/127.28.94.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:35.035223 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:35.036383 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:35.046711 23319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:35.046705 23318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:35.046711 23321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:35.046988 23312 server_base.cc:1034] running on GCE node
I20260806 04:33:35.047513 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:35.048223 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:35.049391 23312 hybrid_clock.cc:648] HybridClock initialized: now 1785990815049370 us; error 32 us; skew 500 ppm
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:35.052374 23312 init.cc:377] Logged in from keytab as kudu/127.28.94.1@KRBTEST.COM (short username kudu)
I20260806 04:33:35.053476 23312 webserver.cc:460] Webserver started at http://127.28.94.1:41233/ using document root <none> and password file <none>
I20260806 04:33:35.054091 23312 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:35.054175 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:35.054404 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:35.056749 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "dffbcd2732fdc9502818422de8de8575"
server_key_iv: "71ad225567f0baa482aaa9525155d9a8"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.057425 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "dffbcd2732fdc9502818422de8de8575"
server_key_iv: "71ad225567f0baa482aaa9525155d9a8"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.061714 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:35.064504 23332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.065940 23312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:35.066085 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "dffbcd2732fdc9502818422de8de8575"
server_key_iv: "71ad225567f0baa482aaa9525155d9a8"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.066216 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:35.080339 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:35.083639 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:35.083851 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:35.084512 23312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:35.085484 23312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:35.085536 23312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.085610 23312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:35.085635 23312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.094880 23312 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:45621
I20260806 04:33:35.094926 23494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:45621 every 8 connection(s)
I20260806 04:33:35.095897 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:35.096454 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 23312
I20260806 04:33:35.096586 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:35.096881 29048 external_mini_cluster.cc:1428] Setting key f5d1e70d18d7e37a02326807c2f4af5f
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.1@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.2@KRBTEST.COM; defaulting to no policy
I20260806 04:33:35.111251 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.098008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:55451 (local address 127.28.94.62:36243)
0806 04:33:35.098237 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.098242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.099555 (+ 1313us) server_negotiation.cc:407] Connection header received
0806 04:33:35.100500 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.100504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.100571 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.100657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:35.102045 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.102619 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.103879 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.104134 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.104241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.107530 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:35.107535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:35.107537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:35.107567 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:35.109039 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.109609 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.109614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.109616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.109734 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.110055 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.110057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.110058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.110269 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:35.110355 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.110908 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.111040 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.2@KRBTEST.COM" created.
I20260806 04:33:35.112236 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.098552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36243 (local address 127.28.94.1:55451)
0806 04:33:35.099372 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.099422 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.100271 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.100857 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.100865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.101316 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:35.101861 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.101877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.102769 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.102772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.103737 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.103750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.104311 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.104316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.104588 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.105328 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:35.105356 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:35.107288 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:35.109185 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.109193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.109209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.109496 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.109827 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.109830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.109832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.109982 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.110352 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:35.110358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:35.110641 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.110839 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.111135 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":171,"threads_started":1}
I20260806 04:33:35.113679 23495 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36243
I20260806 04:33:35.113977 23495 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:35.114662 23495 heartbeater.cc:511] Master 127.28.94.62:36243 requested a full tablet report, sending...
I20260806 04:33:35.116497 23220 ts_manager.cc:194] Registered new tserver with Master: 82501c8a3b0b4c2fb5d091cdf621bcb3 (127.28.94.1:45621)
I20260806 04:33:35.117823 23220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.1@KRBTEST.COM'} at 127.28.94.1:55451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
Entry for principal kudu/127.28.94.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab.
I20260806 04:33:35.129560 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:35.249595 23501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:35.249898 23501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:35.249948 23501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:35.254173 23501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:35.254251 23501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:35.254323 23501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:35.260150 23501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.2.keytab
--principal=kudu/127.28.94.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:35.261451 23501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:35.262595 23501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:35.272998 23508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:35.272990 23511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:35.273185 23501 server_base.cc:1034] running on GCE node
W20260806 04:33:35.272990 23506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:35.273717 23501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:35.274451 23501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:35.275622 23501 hybrid_clock.cc:648] HybridClock initialized: now 1785990815275600 us; error 38 us; skew 500 ppm
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:35.278553 23501 init.cc:377] Logged in from keytab as kudu/127.28.94.2@KRBTEST.COM (short username kudu)
I20260806 04:33:35.279583 23501 webserver.cc:460] Webserver started at http://127.28.94.2:36127/ using document root <none> and password file <none>
I20260806 04:33:35.280195 23501 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:35.280267 23501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:35.280485 23501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:35.282773 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "d5884ec8301646ce8ce184c336144b15"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "274697a16bd9fea535ca380024e002cd"
server_key_iv: "954a6c29acb0e5b96bf9603ace527f38"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.283371 23501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "d5884ec8301646ce8ce184c336144b15"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "274697a16bd9fea535ca380024e002cd"
server_key_iv: "954a6c29acb0e5b96bf9603ace527f38"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.287510 23501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 04:33:35.290227 23521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.291625 23501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:33:35.291771 23501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "d5884ec8301646ce8ce184c336144b15"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "274697a16bd9fea535ca380024e002cd"
server_key_iv: "954a6c29acb0e5b96bf9603ace527f38"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.291891 23501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:35.315783 23501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:35.319432 23501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:35.319635 23501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:35.320281 23501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:35.321231 23501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:35.321283 23501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.321338 23501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:35.321362 23501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.330205 23501 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:46401
I20260806 04:33:35.330225 23682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:46401 every 8 connection(s)
I20260806 04:33:35.331210 23501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:35.338151 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 23501
I20260806 04:33:35.338299 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:35.338614 29048 external_mini_cluster.cc:1428] Setting key 0d6cbd8b41f3d48f1fe0122a0eca28e7
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.2@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:35.346447 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.333501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:42183 (local address 127.28.94.62:36243)
0806 04:33:35.333679 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.333684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.334879 (+ 1195us) server_negotiation.cc:407] Connection header received
0806 04:33:35.335745 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.335748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.335800 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.335874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:35.337264 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.337808 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.339424 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.339668 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.339765 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.342788 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:35.342791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:35.342793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:35.342824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:35.344177 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.344740 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.344743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.344744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.344831 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.345215 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.345218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.345220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.345427 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:35.345521 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.346107 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.346268 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:33:35.347424 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.333985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36243 (local address 127.28.94.2:42183)
0806 04:33:35.334714 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.334760 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.335554 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.336038 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.336045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.336427 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:35.337099 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.337117 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.337975 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.337979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.339282 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.339295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.339838 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.339842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.340021 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.340667 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:35.340690 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:35.342572 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:35.344312 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.344319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.344340 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.344631 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.344952 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.344955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.344957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.345117 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.345535 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:35.345544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:35.345844 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.346025 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.346409 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":131,"threads_started":1}
I20260806 04:33:35.348883 23683 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36243
I20260806 04:33:35.349184 23683 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.94.3@KRBTEST.COM; defaulting to no policy
I20260806 04:33:35.349773 23683 heartbeater.cc:511] Master 127.28.94.62:36243 requested a full tablet report, sending...
I20260806 04:33:35.351027 23220 ts_manager.cc:194] Registered new tserver with Master: d5884ec8301646ce8ce184c336144b15 (127.28.94.2:46401)
I20260806 04:33:35.351703 23220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.2@KRBTEST.COM'} at 127.28.94.2:42183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
Entry for principal kudu/127.28.94.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab.
I20260806 04:33:35.368335 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:35.492300 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:35.492604 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:35.492648 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:35.497032 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:35.497108 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:35.497179 23697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:35.502861 23697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.3.keytab
--principal=kudu/127.28.94.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:35.504083 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:35.505236 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:35.515726 23705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:35.515730 23704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:35.515920 23697 server_base.cc:1034] running on GCE node
W20260806 04:33:35.515727 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:35.516584 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:35.517433 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:35.518628 23697 hybrid_clock.cc:648] HybridClock initialized: now 1785990815518609 us; error 32 us; skew 500 ppm
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:35.521608 23697 init.cc:377] Logged in from keytab as kudu/127.28.94.3@KRBTEST.COM (short username kudu)
I20260806 04:33:35.522680 23697 webserver.cc:460] Webserver started at http://127.28.94.3:38245/ using document root <none> and password file <none>
I20260806 04:33:35.523314 23697 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:35.523396 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:35.523640 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:35.526043 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "939e97dc239d4a7f85ac2f94ef35da65"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "ccbb6af6c0d9237eaba792ead2e52745"
server_key_iv: "65ddd843fcbec1a73ef35f1500c63e73"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.526695 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "939e97dc239d4a7f85ac2f94ef35da65"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "ccbb6af6c0d9237eaba792ead2e52745"
server_key_iv: "65ddd843fcbec1a73ef35f1500c63e73"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.531030 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:35.533994 23718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.535388 23697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:35.535538 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "939e97dc239d4a7f85ac2f94ef35da65"
format_stamp: "Formatted at 2026-08-06 04:33:35 on dist-test-slave-ncp9"
server_key: "ccbb6af6c0d9237eaba792ead2e52745"
server_key_iv: "65ddd843fcbec1a73ef35f1500c63e73"
server_key_version: "encryptionkey@0"
I20260806 04:33:35.535666 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:35.567612 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:35.571154 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:35.571388 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:35.572080 23697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:35.573114 23697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:35.573174 23697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.573237 23697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:35.573264 23697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:35.585539 23697 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:39457
I20260806 04:33:35.585559 23895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:39457 every 8 connection(s)
I20260806 04:33:35.586643 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:35.587464 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 23697
I20260806 04:33:35.587615 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:35.587927 29048 external_mini_cluster.cc:1428] Setting key e69140dceaf30954818db8c0f8cf0d6f
Aug 06 04:33:35 dist-test-slave-ncp9 krb5kdc[23158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990815, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.3@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:35.601097 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.588641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:41157 (local address 127.28.94.62:36243)
0806 04:33:35.588797 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.588801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.590163 (+ 1362us) server_negotiation.cc:407] Connection header received
0806 04:33:35.591183 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.591187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.591249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.591332 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:35.592632 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.593256 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.594511 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.594749 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.594853 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.597689 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:35.597694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:35.597696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:35.597729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:35.598979 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.599524 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.599528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.599529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.599660 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.599942 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.599944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.599946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.600146 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:35.600228 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.600772 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.600954 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:33:35.602126 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.589185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:36243 (local address 127.28.94.3:41157)
0806 04:33:35.589997 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.590048 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.590959 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.591519 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.591527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.591926 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:35.592472 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.592488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.593404 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.593407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.594372 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.594384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.594921 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.594926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.595121 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.595773 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:33:35.595793 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:35.597483 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:35.599097 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.599103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.599118 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.599405 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.599742 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:33:35.599744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:33:35.599745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:33:35.599880 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:33:35.600222 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:35.600227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:35.600507 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.600703 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.601068 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":156,"threads_started":1}
I20260806 04:33:35.603282 23898 heartbeater.cc:347] Connected to a master server at 127.28.94.62:36243
I20260806 04:33:35.603511 23898 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:35.604035 23898 heartbeater.cc:511] Master 127.28.94.62:36243 requested a full tablet report, sending...
I20260806 04:33:35.605125 23220 ts_manager.cc:194] Registered new tserver with Master: 939e97dc239d4a7f85ac2f94ef35da65 (127.28.94.3:39457)
I20260806 04:33:35.605695 23220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.94.3@KRBTEST.COM'} at 127.28.94.3:41157
I20260806 04:33:35.614892 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:35.623966 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.616943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.28.94.62:36243)
0806 04:33:35.617069 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.617072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.617240 (+ 168us) server_negotiation.cc:407] Connection header received
0806 04:33:35.617351 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.617354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.617400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.617471 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:35.618133 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.618646 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.619465 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.619680 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.619748 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.621106 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:35.621109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:35.621111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:35.621141 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:35.622515 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.622934 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.622937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.622938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.623011 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:35.623213 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:35.623215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:35.623216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:35.623449 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:35.623557 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.623729 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.623834 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:33:35.627099 23220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:35.630331 23220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:35.647539 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.640582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:39457 (local address 127.0.0.1:52790)
0806 04:33:35.641285 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.641298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.641382 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.642272 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.642276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.642295 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:35.642395 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.642401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.644135 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.644138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.645757 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.645770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.646980 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.646984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.647105 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.647135 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.647206 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.647362 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":69,"thread_start_us":102,"threads_started":1}
I20260806 04:33:35.647598 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.641160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.28.94.1:45621)
0806 04:33:35.641598 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.641604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.642232 (+ 628us) server_negotiation.cc:407] Connection header received
0806 04:33:35.642315 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.642320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.642498 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.642631 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:35.643035 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.644187 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.645626 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.646507 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.646628 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.646818 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.647323 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.647439 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":131,"threads_started":1}
I20260806 04:33:35.648980 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.640807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.28.94.3:39457)
0806 04:33:35.641787 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.641805 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.641827 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:33:35.641900 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.641905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.642097 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.642546 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:35.642726 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.643991 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.645912 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.646821 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.648442 (+ 1621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.648624 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.648684 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.648797 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":113,"threads_started":1}
I20260806 04:33:35.649044 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.639588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:46401 (local address 127.0.0.1:47942)
0806 04:33:35.640093 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.640170 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.640325 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.641777 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.641784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.641810 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:35.641961 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.641968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.644395 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.644399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.646641 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.646652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.648651 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.648653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.648721 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.648737 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.648797 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.648876 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":1115,"thread_start_us":108,"threads_started":1}
I20260806 04:33:35.649163 23915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.640488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.28.94.2:46401)
0806 04:33:35.641266 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.641281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.641308 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:33:35.641397 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.641403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.641612 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.641744 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:35.642103 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.644251 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.647780 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.648381 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.648449 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.648562 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.648875 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.649025 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":120,"threads_started":1}
I20260806 04:33:35.649587 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.640968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:45621 (local address 127.0.0.1:50842)
0806 04:33:35.641997 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.642011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.642105 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.642786 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.642790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.642809 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:35.642903 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.642909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.644329 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.644332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.645482 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.645494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.646891 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.646895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.647022 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.647072 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.647184 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.649444 (+ 2260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":2158,"thread_start_us":104,"threads_started":1}
I20260806 04:33:35.652583 23584 tablet_service.cc:1467] Processing CreateTablet for tablet 2c553c4850f949a5b1f866fd70e37426 (DEFAULT_TABLE table=test-table [id=1158b0f5211b4c79bded2807fc98d203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:35.653939 23584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c553c4850f949a5b1f866fd70e37426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:35.653615 23398 tablet_service.cc:1467] Processing CreateTablet for tablet 2c553c4850f949a5b1f866fd70e37426 (DEFAULT_TABLE table=test-table [id=1158b0f5211b4c79bded2807fc98d203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:35.654738 23398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c553c4850f949a5b1f866fd70e37426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:35.660976 23921 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Bootstrap starting.
I20260806 04:33:35.662570 23922 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Bootstrap starting.
I20260806 04:33:35.662645 23791 tablet_service.cc:1467] Processing CreateTablet for tablet 2c553c4850f949a5b1f866fd70e37426 (DEFAULT_TABLE table=test-table [id=1158b0f5211b4c79bded2807fc98d203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:35.663595 23921 tablet_bootstrap.cc:654] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:35.663869 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c553c4850f949a5b1f866fd70e37426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:35.664544 23921 log.cc:824] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:35.665328 23922 tablet_bootstrap.cc:654] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:35.666204 23922 log.cc:824] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:35.666941 23921 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: No bootstrap required, opened a new log
I20260806 04:33:35.667165 23921 ts_tablet_manager.cc:1397] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260806 04:33:35.668421 23922 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: No bootstrap required, opened a new log
I20260806 04:33:35.668632 23922 ts_tablet_manager.cc:1397] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 04:33:35.671622 23927 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Bootstrap starting.
I20260806 04:33:35.671986 23921 raft_consensus.cc:348] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.672305 23921 raft_consensus.cc:374] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:35.672381 23921 raft_consensus.cc:729] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82501c8a3b0b4c2fb5d091cdf621bcb3, State: Initialized, Role: FOLLOWER
I20260806 04:33:35.672917 23921 consensus_queue.cc:260] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.672864 23922 raft_consensus.cc:348] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.673187 23922 raft_consensus.cc:374] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:35.673249 23922 raft_consensus.cc:729] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5884ec8301646ce8ce184c336144b15, State: Initialized, Role: FOLLOWER
I20260806 04:33:35.673700 23922 consensus_queue.cc:260] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.674029 23927 tablet_bootstrap.cc:654] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:35.674074 23921 ts_tablet_manager.cc:1428] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260806 04:33:35.674472 23495 heartbeater.cc:503] Master 127.28.94.62:36243 was elected leader, sending a full tablet report...
I20260806 04:33:35.674497 23683 heartbeater.cc:503] Master 127.28.94.62:36243 was elected leader, sending a full tablet report...
I20260806 04:33:35.674678 23922 ts_tablet_manager.cc:1428] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:33:35.675432 23927 log.cc:824] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:35.679229 23927 tablet_bootstrap.cc:492] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: No bootstrap required, opened a new log
I20260806 04:33:35.679488 23927 ts_tablet_manager.cc:1397] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:33:35.682816 23927 raft_consensus.cc:348] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.683111 23927 raft_consensus.cc:374] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:35.683152 23927 raft_consensus.cc:729] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939e97dc239d4a7f85ac2f94ef35da65, State: Initialized, Role: FOLLOWER
I20260806 04:33:35.683629 23927 consensus_queue.cc:260] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.684391 23898 heartbeater.cc:503] Master 127.28.94.62:36243 was elected leader, sending a full tablet report...
I20260806 04:33:35.684762 23927 ts_tablet_manager.cc:1428] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 04:33:35.832581 23687 tablet.cc:2369] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:35.837833 23899 tablet.cc:2369] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:35.847399 23496 tablet.cc:2369] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:35.875339 23937 raft_consensus.cc:482] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:35.875602 23937 raft_consensus.cc:504] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.877424 23937 leader_election.cc:288] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5884ec8301646ce8ce184c336144b15 (127.28.94.2:46401), 82501c8a3b0b4c2fb5d091cdf621bcb3 (127.28.94.1:45621)
I20260806 04:33:35.882275 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.877739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:49345 (local address 127.28.94.1:45621)
0806 04:33:35.877872 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.877877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.878252 (+ 375us) server_negotiation.cc:407] Connection header received
0806 04:33:35.878441 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.878444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.878501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.878577 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:35.878877 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.879595 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.881069 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.881703 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.881822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.881892 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.882052 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.882139 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:33:35.882295 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.877392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:45621 (local address 127.28.94.3:49345)
0806 04:33:35.878162 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.878176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.878327 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.878592 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.878595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.878615 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:35.878759 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.878765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.879722 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.879726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.880841 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.880853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.881832 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.881835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.881915 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.881937 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.882007 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.882092 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":390,"thread_start_us":140,"threads_started":1}
I20260806 04:33:35.882447 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.877384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:46401 (local address 127.28.94.3:33733)
0806 04:33:35.877604 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.877619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:35.877717 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:35.878076 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:35.878079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:35.878101 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:35.878231 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.878240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.879168 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.879171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.880161 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:35.880173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.881959 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.881963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:35.882094 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.882120 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:33:35.882216 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:33:35.882313 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 04:33:35.882804 23915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.877561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:33733 (local address 127.28.94.2:46401)
0806 04:33:35.877719 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.877724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.877743 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:33:35.877805 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.877808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.877862 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.877939 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:35.878358 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.879065 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.880330 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.880972 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.881072 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.881146 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.882571 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.882674 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:33:35.883574 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c553c4850f949a5b1f866fd70e37426" candidate_uuid: "939e97dc239d4a7f85ac2f94ef35da65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" is_pre_election: true
I20260806 04:33:35.883559 23615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c553c4850f949a5b1f866fd70e37426" candidate_uuid: "939e97dc239d4a7f85ac2f94ef35da65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5884ec8301646ce8ce184c336144b15" is_pre_election: true
I20260806 04:33:35.883930 23425 raft_consensus.cc:2454] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939e97dc239d4a7f85ac2f94ef35da65 in term 0.
I20260806 04:33:35.883952 23615 raft_consensus.cc:2454] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939e97dc239d4a7f85ac2f94ef35da65 in term 0.
I20260806 04:33:35.885440 23721 leader_election.cc:302] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 82501c8a3b0b4c2fb5d091cdf621bcb3, 939e97dc239d4a7f85ac2f94ef35da65, d5884ec8301646ce8ce184c336144b15; no voters:
I20260806 04:33:35.885823 23937 raft_consensus.cc:2781] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:35.885918 23937 raft_consensus.cc:482] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:35.885968 23937 raft_consensus.cc:3037] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:35.887349 23937 raft_consensus.cc:504] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.887800 23937 leader_election.cc:288] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [CANDIDATE]: Term 1 election: Requested vote from peers d5884ec8301646ce8ce184c336144b15 (127.28.94.2:46401), 82501c8a3b0b4c2fb5d091cdf621bcb3 (127.28.94.1:45621)
I20260806 04:33:35.888363 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c553c4850f949a5b1f866fd70e37426" candidate_uuid: "939e97dc239d4a7f85ac2f94ef35da65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3"
I20260806 04:33:35.888549 23615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c553c4850f949a5b1f866fd70e37426" candidate_uuid: "939e97dc239d4a7f85ac2f94ef35da65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5884ec8301646ce8ce184c336144b15"
I20260806 04:33:35.888527 23425 raft_consensus.cc:3037] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:35.888698 23615 raft_consensus.cc:3037] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:35.889981 23615 raft_consensus.cc:2454] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939e97dc239d4a7f85ac2f94ef35da65 in term 1.
I20260806 04:33:35.889981 23425 raft_consensus.cc:2454] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939e97dc239d4a7f85ac2f94ef35da65 in term 1.
I20260806 04:33:35.890388 23725 leader_election.cc:302] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82501c8a3b0b4c2fb5d091cdf621bcb3, 939e97dc239d4a7f85ac2f94ef35da65; no voters:
I20260806 04:33:35.890645 23937 raft_consensus.cc:2781] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:35.890908 23937 raft_consensus.cc:686] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 1 LEADER]: Becoming Leader. State: Replica: 939e97dc239d4a7f85ac2f94ef35da65, State: Running, Role: LEADER
I20260806 04:33:35.891319 23937 consensus_queue.cc:237] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } }
I20260806 04:33:35.894929 23219 catalog_manager.cc:5495] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 939e97dc239d4a7f85ac2f94ef35da65 (127.28.94.3). New cstate: current_term: 1 leader_uuid: "939e97dc239d4a7f85ac2f94ef35da65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939e97dc239d4a7f85ac2f94ef35da65" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 39457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 } health_report { overall_health: UNKNOWN } } }
I20260806 04:33:35.941424 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.937765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.28.94.3:39457)
0806 04:33:35.937900 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:35.937906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:35.937925 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:33:35.938078 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:35.938080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:35.938145 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:35.938236 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:35.938429 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.938929 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.939881 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:35.940110 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:35.940182 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:35.940490 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:35.941020 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:35.941121 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:35.941178 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 04:33:35.941238 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:33:35.947821 23425 raft_consensus.cc:1261] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 1 FOLLOWER]: Refusing update from remote peer 939e97dc239d4a7f85ac2f94ef35da65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:35.947821 23615 raft_consensus.cc:1261] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 1 FOLLOWER]: Refusing update from remote peer 939e97dc239d4a7f85ac2f94ef35da65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:35.948585 23940 consensus_queue.cc:1035] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5884ec8301646ce8ce184c336144b15" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:35.948794 23937 consensus_queue.cc:1035] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82501c8a3b0b4c2fb5d091cdf621bcb3" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:35.956396 23944 mvcc.cc:205] Tried to move back new op lower bound from 7315418382114181120 to 7315418381893742592. Current Snapshot: MvccSnapshot[applied={T|T < 7315418382114181120}]
I20260806 04:33:35.957909 23943 mvcc.cc:205] Tried to move back new op lower bound from 7315418382114181120 to 7315418381893742592. Current Snapshot: MvccSnapshot[applied={T|T < 7315418382114181120}]
I20260806 04:33:35.966653 23220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:35.966847 23220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:35.969682 23220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f964aebaaead4c6bbd29e52e06da6c25: Sending DeleteTablet for 3 replicas of tablet 2c553c4850f949a5b1f866fd70e37426
I20260806 04:33:35.970674 23398 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c553c4850f949a5b1f866fd70e37426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:35 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:50842
I20260806 04:33:35.970619 23584 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c553c4850f949a5b1f866fd70e37426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:35 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:47942
I20260806 04:33:35.970805 23791 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c553c4850f949a5b1f866fd70e37426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:35 UTC) from {username='kudu', principal='kudu/127.28.94.62@KRBTEST.COM'} at 127.0.0.1:52790
I20260806 04:33:35.971395 23960 tablet_replica.cc:330] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: stopping tablet replica
I20260806 04:33:35.971593 23961 tablet_replica.cc:330] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3: stopping tablet replica
I20260806 04:33:35.971822 23960 raft_consensus.cc:2229] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:35.972055 23961 raft_consensus.cc:2229] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:35.972117 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 23312
I20260806 04:33:35.972240 23960 raft_consensus.cc:2258] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:35.972515 23961 raft_consensus.cc:2258] T 2c553c4850f949a5b1f866fd70e37426 P 82501c8a3b0b4c2fb5d091cdf621bcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:35.979717 23960 ts_tablet_manager.cc:1905] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:35.982287 23960 ts_tablet_manager.cc:1918] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:35.982417 23960 log.cc:1196] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/2c553c4850f949a5b1f866fd70e37426
I20260806 04:33:35.982761 23960 ts_tablet_manager.cc:1939] T 2c553c4850f949a5b1f866fd70e37426 P d5884ec8301646ce8ce184c336144b15: Deleting consensus metadata
I20260806 04:33:35.982916 23963 tablet_replica.cc:330] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: stopping tablet replica
I20260806 04:33:35.983382 23963 raft_consensus.cc:2229] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:35.983934 23963 raft_consensus.cc:2258] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:35.984351 23198 catalog_manager.cc:4841] TS d5884ec8301646ce8ce184c336144b15 (127.28.94.2:46401): tablet 2c553c4850f949a5b1f866fd70e37426 (table test-table [id=1158b0f5211b4c79bded2807fc98d203]) successfully deleted
I20260806 04:33:35.985607 23963 ts_tablet_manager.cc:1905] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:33:35.987262 23199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:45621 (error 108)
I20260806 04:33:35.988205 23963 ts_tablet_manager.cc:1918] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:35.988312 23963 log.cc:1196] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/2c553c4850f949a5b1f866fd70e37426
I20260806 04:33:35.988492 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:35.988149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:45621 (local address 127.0.0.1:50850)
0806 04:33:35.988303 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:35.988392 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 04:33:35.988571 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 23501
I20260806 04:33:35.988662 23963 ts_tablet_manager.cc:1939] T 2c553c4850f949a5b1f866fd70e37426 P 939e97dc239d4a7f85ac2f94ef35da65: Deleting consensus metadata
W20260806 04:33:35.992300 23199 catalog_manager.cc:4584] TS 82501c8a3b0b4c2fb5d091cdf621bcb3 (127.28.94.1:45621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c553c4850f949a5b1f866fd70e37426: Network error: Client connection negotiation failed: client connection to 127.28.94.1:45621: connect: Connection refused (error 111)
I20260806 04:33:35.992574 23199 catalog_manager.cc:4841] TS 939e97dc239d4a7f85ac2f94ef35da65 (127.28.94.3:39457): tablet 2c553c4850f949a5b1f866fd70e37426 (table test-table [id=1158b0f5211b4c79bded2807fc98d203]) successfully deleted
I20260806 04:33:36.000758 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 23697
I20260806 04:33:36.012357 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 23177
2026-08-06T04:33:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2676 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T04:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:36Z Disabled control of system clock
I20260806 04:33:36.056000 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39753
--webserver_interface=127.28.94.62
--webserver_port=0
--builtin_ntp_servers=127.28.94.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:36.181216 23970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:36.181541 23970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:36.181579 23970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:36.185915 23970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:36.186000 23970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:36.186015 23970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:36.186031 23970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:36.186045 23970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:36.191545 23970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:36.192817 23970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:36.194350 23970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:36.203421 23981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.203418 23979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.203577 23970 server_base.cc:1034] running on GCE node
W20260806 04:33:36.203775 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.204427 23970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:36.205807 23970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:36.207026 23970 hybrid_clock.cc:648] HybridClock initialized: now 1785990816207007 us; error 36 us; skew 500 ppm
I20260806 04:33:36.213557 23970 webserver.cc:460] Webserver started at http://127.28.94.62:43251/ using document root <none> and password file <none>
I20260806 04:33:36.214277 23970 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:36.214350 23970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:36.214581 23970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:36.216952 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "12e4414498a2472ba218b39c7b193929"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "b92cc8d3733e1d960c02d0cbdd4d17e8"
server_key_iv: "80f0f3081d335f584a0485e1d7ab1d73"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.217625 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "12e4414498a2472ba218b39c7b193929"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "b92cc8d3733e1d960c02d0cbdd4d17e8"
server_key_iv: "80f0f3081d335f584a0485e1d7ab1d73"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.221871 23970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:36.224751 23990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.226254 23970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:33:36.226413 23970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "12e4414498a2472ba218b39c7b193929"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "b92cc8d3733e1d960c02d0cbdd4d17e8"
server_key_iv: "80f0f3081d335f584a0485e1d7ab1d73"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.226540 23970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:36.243050 23970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:36.243920 23970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:36.244122 23970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:36.251607 23970 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39753
I20260806 04:33:36.251650 24060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39753 every 8 connection(s)
I20260806 04:33:36.252679 23970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:36.255364 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 23970
I20260806 04:33:36.255508 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:36.255843 29048 external_mini_cluster.cc:1428] Setting key 9306e2f9591437bc2628fae1f7673dc2
I20260806 04:33:36.256523 24061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:36.263847 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: Bootstrap starting.
I20260806 04:33:36.266237 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.257427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.28.94.62:39753)
0806 04:33:36.258157 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.258173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.258212 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:33:36.258906 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.258925 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.259435 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.259855 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:36.260782 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.261856 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.262792 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.263309 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.263474 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.264348 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:36.264353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:36.264369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:36.264793 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:36.264854 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:36.264876 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:36.265007 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.265148 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.265439 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":159,"threads_started":1}
I20260806 04:33:36.266515 24061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:36.267370 24061 log.cc:824] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:36.269764 24061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: No bootstrap required, opened a new log
I20260806 04:33:36.273813 24061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } }
I20260806 04:33:36.274180 24061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:36.274241 24061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e4414498a2472ba218b39c7b193929, State: Initialized, Role: FOLLOWER
I20260806 04:33:36.274685 24061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } }
I20260806 04:33:36.274814 24061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:36.274863 24061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:36.274942 24061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:36.275949 24061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } }
I20260806 04:33:36.276293 24061 leader_election.cc:302] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12e4414498a2472ba218b39c7b193929; no voters:
I20260806 04:33:36.276616 24061 leader_election.cc:288] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:36.276764 24067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:36.277019 24067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [term 1 LEADER]: Becoming Leader. State: Replica: 12e4414498a2472ba218b39c7b193929, State: Running, Role: LEADER
I20260806 04:33:36.277369 24067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } }
I20260806 04:33:36.277985 24061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:36.279141 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12e4414498a2472ba218b39c7b193929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } } }
I20260806 04:33:36.279347 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:36.279140 24071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12e4414498a2472ba218b39c7b193929. Latest consensus state: current_term: 1 leader_uuid: "12e4414498a2472ba218b39c7b193929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12e4414498a2472ba218b39c7b193929" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39753 } } }
I20260806 04:33:36.279841 24071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929 [sys.catalog]: This master's current role is: LEADER
W20260806 04:33:36.283336 24088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:33:36.283437 24088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:33:36.283596 24081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:36.284631 24081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:36.291002 24081 catalog_manager.cc:1347] Generated new cluster ID: 1ca5066b242a4537865b589d4f079f16
I20260806 04:33:36.291101 24081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:36.310634 24081 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:36.312070 24081 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:36.327416 24081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: Generated new TSK 0
I20260806 04:33:36.328431 24081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:36.336413 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.1:0
--local_ip_for_outbound_sockets=127.28.94.1
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--builtin_ntp_servers=127.28.94.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:33:36.458143 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:36.458444 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:36.458484 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:36.462670 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:36.462749 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:36.462807 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.1
I20260806 04:33:36.467905 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.94.1
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.1
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:36.469190 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:36.470674 24095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:36.480930 24109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.480947 24106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.480947 24105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.481235 24095 server_base.cc:1034] running on GCE node
I20260806 04:33:36.481739 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:36.482510 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:36.483721 24095 hybrid_clock.cc:648] HybridClock initialized: now 1785990816483677 us; error 51 us; skew 500 ppm
I20260806 04:33:36.485877 24095 webserver.cc:460] Webserver started at http://127.28.94.1:38775/ using document root <none> and password file <none>
I20260806 04:33:36.486521 24095 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:36.486598 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:36.486812 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:36.489120 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data/instance:
uuid: "704c622793d24a29897270f99c27681f"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "bae78108e5914963a12a2863d0984129"
server_key_iv: "1dae59eece6aa9cdc5245407e28aab6a"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.489804 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance:
uuid: "704c622793d24a29897270f99c27681f"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "bae78108e5914963a12a2863d0984129"
server_key_iv: "1dae59eece6aa9cdc5245407e28aab6a"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.493986 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:33:36.496819 24115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.498194 24095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:36.498332 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal
uuid: "704c622793d24a29897270f99c27681f"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "bae78108e5914963a12a2863d0984129"
server_key_iv: "1dae59eece6aa9cdc5245407e28aab6a"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.498463 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:36.520448 24095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:36.521323 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:36.521550 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:36.522238 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:36.523294 24095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:36.523346 24095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.523404 24095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:36.523429 24095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.532651 24095 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.1:39419
I20260806 04:33:36.532676 24283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.1:39419 every 8 connection(s)
I20260806 04:33:36.533717 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/data/info.pb
I20260806 04:33:36.534711 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 24095
I20260806 04:33:36.534838 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-0/wal/instance
I20260806 04:33:36.535126 29048 external_mini_cluster.cc:1428] Setting key 90cdab22cfbb63498b000249fab26b03
I20260806 04:33:36.537356 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.2:0
--local_ip_for_outbound_sockets=127.28.94.2
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--builtin_ntp_servers=127.28.94.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:36.544850 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.535835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.1:49795 (local address 127.28.94.62:39753)
0806 04:33:36.535992 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.535997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.537240 (+ 1243us) server_negotiation.cc:407] Connection header received
0806 04:33:36.538754 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.538757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.538819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.538920 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:36.540202 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.540844 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.542590 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.542829 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.542917 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.544326 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:36.544329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:36.544331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:36.544348 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:36.544370 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:36.544378 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:36.544435 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.544620 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.544688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:33:36.546208 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.536290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39753 (local address 127.28.94.1:49795)
0806 04:33:36.537094 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.537140 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.538506 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.539131 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.539143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.539271 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:36.540013 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.540032 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.541020 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.541024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.542439 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.542453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.543008 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.543013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.543237 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.543953 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:36.543956 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:36.544021 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:36.544127 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:36.544454 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:36.544464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:36.544472 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.544620 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.545030 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260806 04:33:36.547705 24285 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39753
I20260806 04:33:36.548020 24285 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:36.548813 24285 heartbeater.cc:511] Master 127.28.94.62:39753 requested a full tablet report, sending...
I20260806 04:33:36.550906 24015 ts_manager.cc:194] Registered new tserver with Master: 704c622793d24a29897270f99c27681f (127.28.94.1:39419)
I20260806 04:33:36.552541 24015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.1:49795
W20260806 04:33:36.661020 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:36.661329 24290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:36.661368 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:36.665493 24290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:36.665555 24290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:36.665613 24290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.2
I20260806 04:33:36.670537 24290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.94.2
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.2
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:36.671720 24290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:36.673172 24290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:36.683028 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.683053 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.683049 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.683451 24290 server_base.cc:1034] running on GCE node
I20260806 04:33:36.683914 24290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:36.684610 24290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:36.685779 24290 hybrid_clock.cc:648] HybridClock initialized: now 1785990816685743 us; error 50 us; skew 500 ppm
I20260806 04:33:36.687745 24290 webserver.cc:460] Webserver started at http://127.28.94.2:38375/ using document root <none> and password file <none>
I20260806 04:33:36.688347 24290 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:36.688412 24290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:36.688627 24290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:36.690828 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data/instance:
uuid: "172b29f7100544a5a4322eb73186b596"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "7118c35837b78c83c914694ab504a4bb"
server_key_iv: "db2735f67bc9d184e42dbb2ee2266ae4"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.691408 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance:
uuid: "172b29f7100544a5a4322eb73186b596"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "7118c35837b78c83c914694ab504a4bb"
server_key_iv: "db2735f67bc9d184e42dbb2ee2266ae4"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.695322 24290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:36.697928 24308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.699244 24290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:36.699368 24290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal
uuid: "172b29f7100544a5a4322eb73186b596"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "7118c35837b78c83c914694ab504a4bb"
server_key_iv: "db2735f67bc9d184e42dbb2ee2266ae4"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.699482 24290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:36.718184 24290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:36.718977 24290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:36.719172 24290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:36.719820 24290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:36.720861 24290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:36.720938 24290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.721004 24290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:36.721027 24290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.730304 24290 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.2:38189
I20260806 04:33:36.730327 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.2:38189 every 8 connection(s)
I20260806 04:33:36.731387 24290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/data/info.pb
I20260806 04:33:36.736722 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 24290
I20260806 04:33:36.736869 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal/instance
I20260806 04:33:36.737195 29048 external_mini_cluster.cc:1428] Setting key 5b32e9721d9da6a9e33e43609f2e8e91
I20260806 04:33:36.739279 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.94.3:0
--local_ip_for_outbound_sockets=127.28.94.3
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--builtin_ntp_servers=127.28.94.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:33:36.741698 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.733339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.2:42643 (local address 127.28.94.62:39753)
0806 04:33:36.733474 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.733478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.734751 (+ 1273us) server_negotiation.cc:407] Connection header received
0806 04:33:36.736277 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.736282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.736348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.736424 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:36.737556 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.738100 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.739509 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.739803 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.739918 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.741174 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:36.741177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:36.741180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:36.741194 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:36.741214 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:36.741222 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:36.741281 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.741452 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.741526 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:33:36.742738 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.733814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39753 (local address 127.28.94.2:42643)
0806 04:33:36.734568 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.734619 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.736006 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.736646 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.736655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.736753 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:36.737402 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.737419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.738241 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.738244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.739323 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.739335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.739965 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.739968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.740136 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.740815 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:36.740817 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:36.740883 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:36.741002 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:36.741311 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:36.741323 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:36.741329 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.741447 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.741777 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":140,"threads_started":1}
I20260806 04:33:36.744048 24478 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39753
I20260806 04:33:36.744318 24478 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:36.744998 24478 heartbeater.cc:511] Master 127.28.94.62:39753 requested a full tablet report, sending...
I20260806 04:33:36.746225 24015 ts_manager.cc:194] Registered new tserver with Master: 172b29f7100544a5a4322eb73186b596 (127.28.94.2:38189)
I20260806 04:33:36.746842 24015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.2:42643
W20260806 04:33:36.862488 24482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:36.862797 24482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:36.862850 24482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:36.867343 24482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:36.867415 24482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:36.867496 24482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.94.3
I20260806 04:33:36.872509 24482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.94.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.94.3
--webserver_port=0
--tserver_master_addrs=127.28.94.62:39753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.94.3
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:36.873732 24482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:36.875141 24482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:36.885121 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:36.885131 24497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.885389 24482 server_base.cc:1034] running on GCE node
W20260806 04:33:36.885115 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:36.885869 24482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:36.886510 24482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:36.887673 24482 hybrid_clock.cc:648] HybridClock initialized: now 1785990816887656 us; error 28 us; skew 500 ppm
I20260806 04:33:36.890636 24482 webserver.cc:460] Webserver started at http://127.28.94.3:34807/ using document root <none> and password file <none>
I20260806 04:33:36.891222 24482 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:36.891287 24482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:36.891495 24482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:36.893643 24482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data/instance:
uuid: "df26fc715e0b4b9d9450399a3108fd4c"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "4661adec334211ca383299cef7325167"
server_key_iv: "e0befce76ffb992ad6e8e0651fc29e1f"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.894199 24482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance:
uuid: "df26fc715e0b4b9d9450399a3108fd4c"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "4661adec334211ca383299cef7325167"
server_key_iv: "e0befce76ffb992ad6e8e0651fc29e1f"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.898043 24482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:36.900609 24503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.901889 24482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:36.902016 24482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal
uuid: "df26fc715e0b4b9d9450399a3108fd4c"
format_stamp: "Formatted at 2026-08-06 04:33:36 on dist-test-slave-ncp9"
server_key: "4661adec334211ca383299cef7325167"
server_key_iv: "e0befce76ffb992ad6e8e0651fc29e1f"
server_key_version: "encryptionkey@0"
I20260806 04:33:36.902148 24482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:36.913096 24482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:36.913754 24482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:36.913919 24482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:36.914495 24482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:33:36.915474 24482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:33:36.915521 24482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.915565 24482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:33:36.915594 24482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:36.923965 24482 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.3:38451
I20260806 04:33:36.923990 24661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.3:38451 every 8 connection(s)
I20260806 04:33:36.924880 24482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/data/info.pb
I20260806 04:33:36.928339 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 24482
I20260806 04:33:36.928463 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal/instance
I20260806 04:33:36.928781 29048 external_mini_cluster.cc:1428] Setting key 6c4b87c619683be01218b3e4dd187b4d
I20260806 04:33:36.934194 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.926724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:42977 (local address 127.28.94.62:39753)
0806 04:33:36.926872 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.926877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.927941 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 04:33:36.929700 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.929703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.929763 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.929858 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:36.930759 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.931279 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.932409 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.932623 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.932716 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.933746 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:36.933750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:36.933752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:36.933764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:36.933783 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:36.933790 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:36.933847 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.933985 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.934050 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:33:36.935074 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.927154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:39753 (local address 127.28.94.3:42977)
0806 04:33:36.927800 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.927840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.929487 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.930038 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.930046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.930133 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:33:36.930626 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.930639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.931403 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.931406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.932294 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.932304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.932772 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.932776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.932962 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.933502 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:33:36.933503 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:33:36.933541 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:33:36.933606 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:33:36.933858 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:33:36.933867 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:33:36.933873 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.933978 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.934270 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260806 04:33:36.936129 24662 heartbeater.cc:347] Connected to a master server at 127.28.94.62:39753
I20260806 04:33:36.936362 24662 heartbeater.cc:464] Registering TS with master...
I20260806 04:33:36.936878 24662 heartbeater.cc:511] Master 127.28.94.62:39753 requested a full tablet report, sending...
I20260806 04:33:36.937970 24015 ts_manager.cc:194] Registered new tserver with Master: df26fc715e0b4b9d9450399a3108fd4c (127.28.94.3:38451)
I20260806 04:33:36.938529 24015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.94.3:42977
I20260806 04:33:36.942638 29048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:33:36.948010 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.944590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.28.94.62:39753)
0806 04:33:36.944713 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.944716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.944944 (+ 228us) server_negotiation.cc:407] Connection header received
0806 04:33:36.945045 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.945047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.945092 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.945158 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:36.945365 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.945896 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.946641 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.946840 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.946902 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.947670 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:36.947673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:36.947676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:36.947688 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:36.947707 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:36.947714 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:36.947770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.947842 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.947891 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:33:36.950814 24015 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:33:36.954224 24015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:33:36.968297 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.962446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:38189 (local address 127.0.0.1:53482)
0806 04:33:36.963101 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.963149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.963302 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.963919 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.963925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.963949 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:36.964090 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.964096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965518 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.965521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.966855 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.966866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.967754 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.967758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.967864 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.967910 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.968013 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.968139 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":79,"thread_start_us":85,"threads_started":1}
I20260806 04:33:36.968317 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.962589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.28.94.2:38189)
0806 04:33:36.962938 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.962949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.963444 (+ 495us) server_negotiation.cc:407] Connection header received
0806 04:33:36.963533 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.963538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.963766 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.963924 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:36.964235 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965404 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.966961 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.967631 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.967738 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.967935 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.968042 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.968142 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":135,"threads_started":1}
I20260806 04:33:36.968852 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.963153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39419 (local address 127.0.0.1:36668)
0806 04:33:36.964264 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.964275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.964360 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.964755 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.964758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.964775 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:36.964869 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.964876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965960 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.965963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.967196 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.967207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.968422 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.968426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.968515 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.968540 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.968614 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.968693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":27,"thread_start_us":80,"threads_started":1}
I20260806 04:33:36.968961 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.963350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.28.94.1:39419)
0806 04:33:36.963791 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.963803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.964370 (+ 567us) server_negotiation.cc:407] Connection header received
0806 04:33:36.964455 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.964461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.964640 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.964774 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:36.964984 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965854 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.967325 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.968083 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.968176 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.968351 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.968687 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.968783 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":110,"threads_started":1}
I20260806 04:33:36.968852 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.962581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.3:38451 (local address 127.0.0.1:42810)
0806 04:33:36.963099 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:36.963152 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:36.963303 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:36.964456 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:36.964458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:36.964472 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:36.964574 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.964579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965832 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.965836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.967200 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:36.967207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.968485 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.968489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:36.968562 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.968580 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 04:33:36.968639 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 04:33:36.968692 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":117,"spinlock_wait_cycles":57088,"thread_start_us":117,"threads_started":1}
I20260806 04:33:36.971460 24179 tablet_service.cc:1467] Processing CreateTablet for tablet 403e7b349bad48c59e301989ed2d0a97 (DEFAULT_TABLE table=test-table [id=a8a48d5e64154d9d9422ea77d886ed47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:36.971457 24388 tablet_service.cc:1467] Processing CreateTablet for tablet 403e7b349bad48c59e301989ed2d0a97 (DEFAULT_TABLE table=test-table [id=a8a48d5e64154d9d9422ea77d886ed47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:36.972359 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e7b349bad48c59e301989ed2d0a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:36.972359 24388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e7b349bad48c59e301989ed2d0a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:36.973623 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:36.963556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.28.94.3:38451)
0806 04:33:36.963969 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:36.963981 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:36.964004 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:33:36.964076 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:36.964081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:36.964265 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:36.964405 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:36.964681 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.965721 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.967339 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:36.967929 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:36.968035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:36.968220 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:36.973185 (+ 4965us) server_negotiation.cc:299] Negotiation successful
0806 04:33:36.973296 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":122,"threads_started":1}
I20260806 04:33:36.976464 24577 tablet_service.cc:1467] Processing CreateTablet for tablet 403e7b349bad48c59e301989ed2d0a97 (DEFAULT_TABLE table=test-table [id=a8a48d5e64154d9d9422ea77d886ed47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:33:36.977576 24577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 403e7b349bad48c59e301989ed2d0a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:36.979305 24685 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Bootstrap starting.
I20260806 04:33:36.980754 24686 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Bootstrap starting.
I20260806 04:33:36.981370 24685 tablet_bootstrap.cc:654] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:36.982055 24685 log.cc:824] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:36.982909 24686 tablet_bootstrap.cc:654] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:36.983755 24686 log.cc:824] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:36.983824 24685 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: No bootstrap required, opened a new log
I20260806 04:33:36.984009 24685 ts_tablet_manager.cc:1397] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 04:33:36.985235 24693 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Bootstrap starting.
I20260806 04:33:36.985667 24686 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: No bootstrap required, opened a new log
I20260806 04:33:36.985842 24686 ts_tablet_manager.cc:1397] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:33:36.987291 24693 tablet_bootstrap.cc:654] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:36.988080 24693 log.cc:824] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:36.988149 24685 raft_consensus.cc:348] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.988418 24685 raft_consensus.cc:374] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:36.988472 24685 raft_consensus.cc:729] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704c622793d24a29897270f99c27681f, State: Initialized, Role: FOLLOWER
I20260806 04:33:36.988842 24685 consensus_queue.cc:260] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.988996 24686 raft_consensus.cc:348] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.989243 24686 raft_consensus.cc:374] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:36.989300 24686 raft_consensus.cc:729] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172b29f7100544a5a4322eb73186b596, State: Initialized, Role: FOLLOWER
I20260806 04:33:36.989466 24285 heartbeater.cc:503] Master 127.28.94.62:39753 was elected leader, sending a full tablet report...
I20260806 04:33:36.989687 24686 consensus_queue.cc:260] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.990293 24478 heartbeater.cc:503] Master 127.28.94.62:39753 was elected leader, sending a full tablet report...
I20260806 04:33:36.990621 24686 ts_tablet_manager.cc:1428] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:33:36.990712 24685 ts_tablet_manager.cc:1428] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:33:36.992384 24693 tablet_bootstrap.cc:492] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: No bootstrap required, opened a new log
I20260806 04:33:36.992558 24693 ts_tablet_manager.cc:1397] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260806 04:33:36.995875 24693 raft_consensus.cc:348] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.996071 24693 raft_consensus.cc:374] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:36.996109 24693 raft_consensus.cc:729] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df26fc715e0b4b9d9450399a3108fd4c, State: Initialized, Role: FOLLOWER
I20260806 04:33:36.996479 24693 consensus_queue.cc:260] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:36.997100 24662 heartbeater.cc:503] Master 127.28.94.62:39753 was elected leader, sending a full tablet report...
I20260806 04:33:36.997311 24693 ts_tablet_manager.cc:1428] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260806 04:33:37.035197 24287 tablet.cc:2369] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:37.175959 24664 tablet.cc:2369] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:33:37.232635 24479 tablet.cc:2369] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:33:37.240432 24699 raft_consensus.cc:482] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:33:37.240685 24699 raft_consensus.cc:504] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:37.242049 24699 leader_election.cc:288] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 172b29f7100544a5a4322eb73186b596 (127.28.94.2:38189), 704c622793d24a29897270f99c27681f (127.28.94.1:39419)
I20260806 04:33:37.246098 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.242378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.2:38189 (local address 127.28.94.3:48959)
0806 04:33:37.242617 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:37.242634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:37.242752 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:37.242996 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:37.242999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:37.243014 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:37.243171 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:37.243179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.244020 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.244024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:37.244878 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:37.244923 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.245660 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:37.245766 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:37.245786 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:33:37.245853 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:33:37.245938 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260806 04:33:37.246095 24679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.242511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:48959 (local address 127.28.94.2:38189)
0806 04:33:37.242648 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:37.242653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:37.242691 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:33:37.242863 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:37.242865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:37.242912 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:37.242981 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:37.243313 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.243907 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245043 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.245527 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245633 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:37.245699 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:37.245885 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 04:33:37.245961 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:33:37.246146 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.242491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.94.3:46481 (local address 127.28.94.1:39419)
0806 04:33:37.242643 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:37.242648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:37.242963 (+ 315us) server_negotiation.cc:407] Connection header received
0806 04:33:37.243096 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:37.243098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:37.243164 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:37.243237 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:33:37.243533 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.244052 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245168 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.245594 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245705 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:37.245768 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:37.245944 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:33:37.246018 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:33:37.246167 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.242378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39419 (local address 127.28.94.3:46481)
0806 04:33:37.242850 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:37.242862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:33:37.242987 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:33:37.243263 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:33:37.243265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:33:37.243286 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:33:37.243399 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:37.243405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.244160 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.244163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:37.244996 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:33:37.245011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.245729 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.245733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:33:37.245832 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:37.245853 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:33:37.245918 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:33:37.245987 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":53,"thread_start_us":149,"threads_started":1}
I20260806 04:33:37.246901 24216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403e7b349bad48c59e301989ed2d0a97" candidate_uuid: "df26fc715e0b4b9d9450399a3108fd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704c622793d24a29897270f99c27681f" is_pre_election: true
I20260806 04:33:37.246963 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403e7b349bad48c59e301989ed2d0a97" candidate_uuid: "df26fc715e0b4b9d9450399a3108fd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172b29f7100544a5a4322eb73186b596" is_pre_election: true
I20260806 04:33:37.247262 24410 raft_consensus.cc:2454] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df26fc715e0b4b9d9450399a3108fd4c in term 0.
I20260806 04:33:37.247264 24216 raft_consensus.cc:2454] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df26fc715e0b4b9d9450399a3108fd4c in term 0.
I20260806 04:33:37.247833 24509 leader_election.cc:302] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704c622793d24a29897270f99c27681f, df26fc715e0b4b9d9450399a3108fd4c; no voters:
I20260806 04:33:37.248288 24699 raft_consensus.cc:2781] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:33:37.248381 24699 raft_consensus.cc:482] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:33:37.248413 24699 raft_consensus.cc:3037] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:37.249562 24699 raft_consensus.cc:504] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:37.249944 24699 leader_election.cc:288] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [CANDIDATE]: Term 1 election: Requested vote from peers 172b29f7100544a5a4322eb73186b596 (127.28.94.2:38189), 704c622793d24a29897270f99c27681f (127.28.94.1:39419)
I20260806 04:33:37.250389 24216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403e7b349bad48c59e301989ed2d0a97" candidate_uuid: "df26fc715e0b4b9d9450399a3108fd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704c622793d24a29897270f99c27681f"
I20260806 04:33:37.250389 24410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "403e7b349bad48c59e301989ed2d0a97" candidate_uuid: "df26fc715e0b4b9d9450399a3108fd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "172b29f7100544a5a4322eb73186b596"
I20260806 04:33:37.250550 24410 raft_consensus.cc:3037] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:37.250550 24216 raft_consensus.cc:3037] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:37.251919 24410 raft_consensus.cc:2454] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df26fc715e0b4b9d9450399a3108fd4c in term 1.
I20260806 04:33:37.251919 24216 raft_consensus.cc:2454] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df26fc715e0b4b9d9450399a3108fd4c in term 1.
I20260806 04:33:37.252305 24504 leader_election.cc:302] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 172b29f7100544a5a4322eb73186b596, df26fc715e0b4b9d9450399a3108fd4c; no voters:
I20260806 04:33:37.252599 24699 raft_consensus.cc:2781] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:37.252791 24699 raft_consensus.cc:686] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 1 LEADER]: Becoming Leader. State: Replica: df26fc715e0b4b9d9450399a3108fd4c, State: Running, Role: LEADER
I20260806 04:33:37.253441 24699 consensus_queue.cc:237] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } }
I20260806 04:33:37.257229 24015 catalog_manager.cc:5495] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to df26fc715e0b4b9d9450399a3108fd4c (127.28.94.3). New cstate: current_term: 1 leader_uuid: "df26fc715e0b4b9d9450399a3108fd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df26fc715e0b4b9d9450399a3108fd4c" member_type: VOTER last_known_addr { host: "127.28.94.3" port: 38451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 } health_report { overall_health: UNKNOWN } } }
I20260806 04:33:37.332710 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.329250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.28.94.3:38451)
0806 04:33:37.329381 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:37.329386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:37.329400 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:33:37.329493 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:37.329495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:37.329552 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:37.329646 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:33:37.329829 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.330364 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.331255 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:37.331475 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:37.331559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:37.331837 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:33:37.332309 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:33:37.332401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:37.332467 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 04:33:37.332523 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:33:37.339425 24410 raft_consensus.cc:1261] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 1 FOLLOWER]: Refusing update from remote peer df26fc715e0b4b9d9450399a3108fd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:37.339425 24216 raft_consensus.cc:1261] T 403e7b349bad48c59e301989ed2d0a97 P 704c622793d24a29897270f99c27681f [term 1 FOLLOWER]: Refusing update from remote peer df26fc715e0b4b9d9450399a3108fd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:33:37.340202 24701 consensus_queue.cc:1035] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "172b29f7100544a5a4322eb73186b596" member_type: VOTER last_known_addr { host: "127.28.94.2" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:37.340409 24699 consensus_queue.cc:1035] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "704c622793d24a29897270f99c27681f" member_type: VOTER last_known_addr { host: "127.28.94.1" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:33:37.348979 24706 mvcc.cc:205] Tried to move back new op lower bound from 7315418387813720064 to 7315418387472945152. Current Snapshot: MvccSnapshot[applied={T|T < 7315418387813720064}]
I20260806 04:33:37.350979 24708 mvcc.cc:205] Tried to move back new op lower bound from 7315418387813720064 to 7315418387472945152. Current Snapshot: MvccSnapshot[applied={T|T < 7315418387813720064 or (T in {7315418387813720064})}]
I20260806 04:33:37.359679 24014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:37.359874 24014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:33:37.362877 24014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12e4414498a2472ba218b39c7b193929: Sending DeleteTablet for 3 replicas of tablet 403e7b349bad48c59e301989ed2d0a97
I20260806 04:33:37.363895 24388 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e7b349bad48c59e301989ed2d0a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:37 UTC) from {username='slave'} at 127.0.0.1:53482
I20260806 04:33:37.363940 24179 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e7b349bad48c59e301989ed2d0a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:37 UTC) from {username='slave'} at 127.0.0.1:36668
I20260806 04:33:37.364010 24577 tablet_service.cc:1514] Processing DeleteTablet for tablet 403e7b349bad48c59e301989ed2d0a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:33:37 UTC) from {username='slave'} at 127.0.0.1:42810
I20260806 04:33:37.364490 24719 tablet_replica.cc:330] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: stopping tablet replica
I20260806 04:33:37.364915 24719 raft_consensus.cc:2229] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:33:37.364974 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 24095
I20260806 04:33:37.365360 24719 raft_consensus.cc:2258] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:37.366323 24721 tablet_replica.cc:330] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: stopping tablet replica
I20260806 04:33:37.366683 24721 raft_consensus.cc:2229] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:33:37.367166 24721 raft_consensus.cc:2258] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:33:37.370352 24719 ts_tablet_manager.cc:1905] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:37.370906 24721 ts_tablet_manager.cc:1905] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:33:37.373828 24721 ts_tablet_manager.cc:1918] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:37.373961 24721 log.cc:1196] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-2/wal/wals/403e7b349bad48c59e301989ed2d0a97
I20260806 04:33:37.374298 24721 ts_tablet_manager.cc:1939] T 403e7b349bad48c59e301989ed2d0a97 P df26fc715e0b4b9d9450399a3108fd4c: Deleting consensus metadata
I20260806 04:33:37.374615 24719 ts_tablet_manager.cc:1918] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:33:37.374969 24719 log.cc:1196] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Deleting WAL directory at /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785990554567992-29048-0/minicluster-data/ts-1/wal/wals/403e7b349bad48c59e301989ed2d0a97
I20260806 04:33:37.375345 24719 ts_tablet_manager.cc:1939] T 403e7b349bad48c59e301989ed2d0a97 P 172b29f7100544a5a4322eb73186b596: Deleting consensus metadata
I20260806 04:33:37.375819 23994 catalog_manager.cc:4841] TS df26fc715e0b4b9d9450399a3108fd4c (127.28.94.3:38451): tablet 403e7b349bad48c59e301989ed2d0a97 (table test-table [id=a8a48d5e64154d9d9422ea77d886ed47]) successfully deleted
I20260806 04:33:37.376730 23991 catalog_manager.cc:4841] TS 172b29f7100544a5a4322eb73186b596 (127.28.94.2:38189): tablet 403e7b349bad48c59e301989ed2d0a97 (table test-table [id=a8a48d5e64154d9d9422ea77d886ed47]) successfully deleted
W20260806 04:33:37.387486 23994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.94.1:39419 (error 108)
I20260806 04:33:37.388178 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 24290
I20260806 04:33:37.410903 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:37.410457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.1:39419 (local address 127.0.0.1:36678)
0806 04:33:37.410681 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 04:33:37.410791 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 04:33:37.411290 23994 catalog_manager.cc:4584] TS 704c622793d24a29897270f99c27681f (127.28.94.1:39419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 403e7b349bad48c59e301989ed2d0a97: Network error: Client connection negotiation failed: client connection to 127.28.94.1:39419: connect: Connection refused (error 111)
I20260806 04:33:37.417229 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 24482
I20260806 04:33:37.431079 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 23970
2026-08-06T04:33:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1420 ms)
[----------] 6 tests from EncryptionPolicyTest (11843 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:37 dist-test-slave-ncp9 krb5kdc[24730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:37 dist-test-slave-ncp9 krb5kdc[24730](info): set up 2 sockets
Aug 06 04:33:37 dist-test-slave-ncp9 krb5kdc[24730](info): commencing operation
krb5kdc: starting...
W20260806 04:33:38.569015 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:38 dist-test-slave-ncp9 krb5kdc[24730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:38.705009 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:38.827040 24750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:38.827365 24750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:38.827409 24750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:38.831951 24750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:38.832060 24750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:38.832082 24750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:38.832099 24750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:38.832129 24750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:38.838030 24750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:38.839265 24750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:38.840401 24750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:38.849514 24757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:38.849525 24758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:38.849525 24760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:38.849781 24750 server_base.cc:1034] running on GCE node
I20260806 04:33:38.850549 24750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:38.851826 24750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:38.852967 24750 hybrid_clock.cc:648] HybridClock initialized: now 1785990818852949 us; error 35 us; skew 500 ppm
Aug 06 04:33:38 dist-test-slave-ncp9 krb5kdc[24730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:38.855932 24750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:38.856971 24750 webserver.cc:460] Webserver started at http://127.0.0.1:37817/ using document root <none> and password file <none>
I20260806 04:33:38.857555 24750 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:38.857618 24750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:38.857805 24750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:38.859997 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "a3c09566d6e24e12a3ba8fd08114dc81"
format_stamp: "Formatted at 2026-08-06 04:33:38 on dist-test-slave-ncp9"
server_key: "a4b9a8f4bd50fd03cb8823cbe3ec150e"
server_key_iv: "dbfd1b76470992aee515ffeaa2745609"
server_key_version: "encryptionkey@0"
I20260806 04:33:38.860576 24750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "a3c09566d6e24e12a3ba8fd08114dc81"
format_stamp: "Formatted at 2026-08-06 04:33:38 on dist-test-slave-ncp9"
server_key: "a4b9a8f4bd50fd03cb8823cbe3ec150e"
server_key_iv: "dbfd1b76470992aee515ffeaa2745609"
server_key_version: "encryptionkey@0"
I20260806 04:33:38.864679 24750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:33:38.867380 24771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:38.868745 24750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:33:38.868872 24750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "a3c09566d6e24e12a3ba8fd08114dc81"
format_stamp: "Formatted at 2026-08-06 04:33:38 on dist-test-slave-ncp9"
server_key: "a4b9a8f4bd50fd03cb8823cbe3ec150e"
server_key_iv: "dbfd1b76470992aee515ffeaa2745609"
server_key_version: "encryptionkey@0"
I20260806 04:33:38.869025 24750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:38.891736 24750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:38.895469 24750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:38.895649 24750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:38.903301 24750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39813
I20260806 04:33:38.903355 24855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39813 every 8 connection(s)
I20260806 04:33:38.904361 24750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:38.907476 24857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:38.913827 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 24750
I20260806 04:33:38.913831 24857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81: Bootstrap starting.
I20260806 04:33:38.913959 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:38.914251 29048 external_mini_cluster.cc:1428] Setting key 8e9382de977ad729e1a209e1c9c63f24
I20260806 04:33:38.916635 24857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:38.917475 24857 log.cc:824] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:38.919521 24857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81: No bootstrap required, opened a new log
Aug 06 04:33:38 dist-test-slave-ncp9 krb5kdc[24730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:33:38.923621 24857 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } }
I20260806 04:33:38.923906 24857 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:38.923978 24857 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c09566d6e24e12a3ba8fd08114dc81, State: Initialized, Role: FOLLOWER
I20260806 04:33:38.924475 24857 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } }
I20260806 04:33:38.924628 24857 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:38.924698 24857 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:38.924806 24857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:38.925904 24857 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } }
I20260806 04:33:38.926290 24857 leader_election.cc:302] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3c09566d6e24e12a3ba8fd08114dc81; no voters:
I20260806 04:33:38.926640 24857 leader_election.cc:288] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:38.926735 24862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:38.926926 24862 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [term 1 LEADER]: Becoming Leader. State: Replica: a3c09566d6e24e12a3ba8fd08114dc81, State: Running, Role: LEADER
I20260806 04:33:38.927258 24862 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } }
I20260806 04:33:38.928117 24857 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:38.929203 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3c09566d6e24e12a3ba8fd08114dc81. Latest consensus state: current_term: 1 leader_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } } }
I20260806 04:33:38.929721 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:38.929739 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:38.915678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60423 (local address 127.0.0.1:39813)
0806 04:33:38.916148 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:38.916171 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:38.916351 (+ 180us) server_negotiation.cc:407] Connection header received
0806 04:33:38.917237 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:38.917259 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:38.917653 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:38.917998 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:38.918827 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:38.919869 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:38.920721 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:38.921262 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:38.921426 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:38.924274 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:38.924279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:38.924295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:38.924331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:38.926104 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:38.926685 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:38.926692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:38.926700 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:38.926863 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:38.927293 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:38.927296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:38.927298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:38.927760 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:38.927970 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:38.928328 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 04:33:38.928584 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260806 04:33:38.929520 24867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c09566d6e24e12a3ba8fd08114dc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39813 } } }
I20260806 04:33:38.930107 24867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:38.930727 24891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:38.933111 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:38.933974 24891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:38.940142 24891 catalog_manager.cc:1347] Generated new cluster ID: eb97771ed46348ba92fe5ba27f1102ab
I20260806 04:33:38.940234 24891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:38.945653 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:38.935521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40837 (local address 127.0.0.1:39813)
0806 04:33:38.935635 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:38.935640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:38.935851 (+ 211us) server_negotiation.cc:407] Connection header received
0806 04:33:38.935996 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:38.935999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:38.936053 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:38.936140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:38.936988 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:38.937714 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:38.939484 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:38.939828 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:38.939933 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:38.941724 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:38.941728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:38.941731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:38.941761 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:38.943404 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:38.943975 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:38.943978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:38.943980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:38.944076 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:38.944402 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:38.944404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:38.944406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:38.944609 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:38.944735 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:38.945163 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 04:33:38.945474 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:33:38.953958 24891 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:38.955273 24891 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:38.959424 24891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3c09566d6e24e12a3ba8fd08114dc81: Generated new TSK 0
I20260806 04:33:38.960220 24891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:38.972129 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 24750
W20260806 04:33:38.983464 24726 connection.cc:535] client connection to 127.0.0.1:39813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39813): Connection reset by peer (error 104)
2026-08-06T04:33:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:39 dist-test-slave-ncp9 krb5kdc[24907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:39 dist-test-slave-ncp9 krb5kdc[24907](info): set up 2 sockets
Aug 06 04:33:39 dist-test-slave-ncp9 krb5kdc[24907](info): commencing operation
krb5kdc: starting...
W20260806 04:33:40.077672 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:40 dist-test-slave-ncp9 krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:40.212281 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:40.332934 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:40.333271 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:40.333333 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:40.337841 24927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:40.337929 24927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:40.337957 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:40.337980 24927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:40.338006 24927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:40.343793 24927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:40.345090 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:40.346206 24927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:40.355257 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:40.355275 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:40.355387 24927 server_base.cc:1034] running on GCE node
W20260806 04:33:40.355270 24936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:40.356235 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:40.357532 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:40.358721 24927 hybrid_clock.cc:648] HybridClock initialized: now 1785990820358703 us; error 33 us; skew 500 ppm
Aug 06 04:33:40 dist-test-slave-ncp9 krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:40.361811 24927 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:40.362877 24927 webserver.cc:460] Webserver started at http://127.0.0.1:40745/ using document root <none> and password file <none>
I20260806 04:33:40.363462 24927 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:40.363525 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:40.363723 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:40.366030 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "614cd579c1f24dbd9fc465596b172a3b"
format_stamp: "Formatted at 2026-08-06 04:33:40 on dist-test-slave-ncp9"
server_key: "40167ba52545d53e6f9f4d6112403755"
server_key_iv: "023321ecc5a438c130fab8d741bf4c5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:40.366606 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "614cd579c1f24dbd9fc465596b172a3b"
format_stamp: "Formatted at 2026-08-06 04:33:40 on dist-test-slave-ncp9"
server_key: "40167ba52545d53e6f9f4d6112403755"
server_key_iv: "023321ecc5a438c130fab8d741bf4c5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:40.370812 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:40.373639 24948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:40.375053 24927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:40.375200 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "614cd579c1f24dbd9fc465596b172a3b"
format_stamp: "Formatted at 2026-08-06 04:33:40 on dist-test-slave-ncp9"
server_key: "40167ba52545d53e6f9f4d6112403755"
server_key_iv: "023321ecc5a438c130fab8d741bf4c5f"
server_key_version: "encryptionkey@0"
I20260806 04:33:40.375327 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:40.388803 24927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:40.392246 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:40.392452 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:40.399860 24927 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39805
I20260806 04:33:40.399879 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39805 every 8 connection(s)
I20260806 04:33:40.400977 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:40.404160 25032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:40.410410 25032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b: Bootstrap starting.
I20260806 04:33:40.410513 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 24927
I20260806 04:33:40.410638 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:40.410928 29048 external_mini_cluster.cc:1428] Setting key 6a3c518f0f6fff1445b5674b386a1d7f
I20260806 04:33:40.413317 25032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:40.414143 25032 log.cc:824] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:40.416278 25032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b: No bootstrap required, opened a new log
I20260806 04:33:40.420307 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:40.412405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45863 (local address 127.0.0.1:39805)
0806 04:33:40.412941 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:40.412960 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:40.413002 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:33:40.413747 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:40.413770 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:40.414152 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:40.414501 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:40.414963 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:40.415958 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:40.416878 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:40.417419 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:40.417578 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:40.418461 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:40.418467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:40.418483 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:40.418917 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:40.418980 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:40.419005 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:40.419124 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:40.419321 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 04:33:40.419478 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":148,"threads_started":1}
I20260806 04:33:40.420176 25032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260806 04:33:40.420431 25032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:40.420486 25032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614cd579c1f24dbd9fc465596b172a3b, State: Initialized, Role: FOLLOWER
I20260806 04:33:40.421012 25032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260806 04:33:40.421166 25032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:40.421233 25032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:40.421335 25032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:40.422406 25032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260806 04:33:40.422807 25032 leader_election.cc:302] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 614cd579c1f24dbd9fc465596b172a3b; no voters:
I20260806 04:33:40.423177 25032 leader_election.cc:288] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:40.423264 25041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:40.423457 25041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [term 1 LEADER]: Becoming Leader. State: Replica: 614cd579c1f24dbd9fc465596b172a3b, State: Running, Role: LEADER
I20260806 04:33:40.423525 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:40.423784 25041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } }
I20260806 04:33:40.424639 25032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:40.425618 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "614cd579c1f24dbd9fc465596b172a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } } }
I20260806 04:33:40.425758 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:40.426149 25041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 614cd579c1f24dbd9fc465596b172a3b. Latest consensus state: current_term: 1 leader_uuid: "614cd579c1f24dbd9fc465596b172a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614cd579c1f24dbd9fc465596b172a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39805 } } }
I20260806 04:33:40.426255 25041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:40.426590 25060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:40.430441 25060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:40.431802 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:40.425817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45219 (local address 127.0.0.1:39805)
0806 04:33:40.425933 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:40.425938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:40.427389 (+ 1451us) server_negotiation.cc:407] Connection header received
0806 04:33:40.427509 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:40.427513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:40.427563 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:40.427645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:40.427963 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:40.428713 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:40.429904 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:40.430231 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:40.430342 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:40.431303 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:40.431307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:40.431309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:40.431326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:40.431357 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:40.431373 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:40.431458 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:40.431619 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 04:33:40.431662 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:33:40.437397 25060 catalog_manager.cc:1347] Generated new cluster ID: 58792cab9d6543238726e9bf49669ebd
I20260806 04:33:40.437484 25060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:40.448365 25060 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:40.449461 25060 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:40.457202 25060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 614cd579c1f24dbd9fc465596b172a3b: Generated new TSK 0
I20260806 04:33:40.458179 25060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:40.467348 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 24927
W20260806 04:33:40.484074 24904 connection.cc:535] client connection to 127.0.0.1:39805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39805): Connection reset by peer (error 104)
2026-08-06T04:33:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:40 dist-test-slave-ncp9 krb5kdc[25082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:40 dist-test-slave-ncp9 krb5kdc[25082](info): set up 2 sockets
Aug 06 04:33:40 dist-test-slave-ncp9 krb5kdc[25082](info): commencing operation
krb5kdc: starting...
W20260806 04:33:41.590343 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:41 dist-test-slave-ncp9 krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:41.721303 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:41.841536 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:41.841850 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:41.841894 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:41.846055 25099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:41.846128 25099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:41.846146 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:41.846164 25099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:41.846179 25099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:41.851897 25099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:41.853163 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:41.854254 25099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:41.863209 25110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:41.863226 25109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:41.863209 25114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:41.863579 25099 server_base.cc:1034] running on GCE node
I20260806 04:33:41.864248 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:41.865545 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:41.866768 25099 hybrid_clock.cc:648] HybridClock initialized: now 1785990821866711 us; error 60 us; skew 500 ppm
Aug 06 04:33:41 dist-test-slave-ncp9 krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:41.872471 25099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:41.873634 25099 webserver.cc:460] Webserver started at http://127.0.0.1:35815/ using document root <none> and password file <none>
I20260806 04:33:41.874255 25099 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:41.874330 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:41.874557 25099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:41.876806 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "7ee72019170f41eca254edd28084f6fa"
format_stamp: "Formatted at 2026-08-06 04:33:41 on dist-test-slave-ncp9"
server_key: "8f22e5c4e061c4bd34697e5ad171c984"
server_key_iv: "e8fd286e93a252a0816a1cd66736b671"
server_key_version: "encryptionkey@0"
I20260806 04:33:41.877460 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "7ee72019170f41eca254edd28084f6fa"
format_stamp: "Formatted at 2026-08-06 04:33:41 on dist-test-slave-ncp9"
server_key: "8f22e5c4e061c4bd34697e5ad171c984"
server_key_iv: "e8fd286e93a252a0816a1cd66736b671"
server_key_version: "encryptionkey@0"
I20260806 04:33:41.881721 25099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:33:41.884377 25125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:41.885756 25099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:41.885902 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "7ee72019170f41eca254edd28084f6fa"
format_stamp: "Formatted at 2026-08-06 04:33:41 on dist-test-slave-ncp9"
server_key: "8f22e5c4e061c4bd34697e5ad171c984"
server_key_iv: "e8fd286e93a252a0816a1cd66736b671"
server_key_version: "encryptionkey@0"
I20260806 04:33:41.886022 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:41.891307 25099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:41.894515 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:41.894724 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:41.902127 25099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44617
I20260806 04:33:41.902138 25193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44617 every 8 connection(s)
I20260806 04:33:41.903172 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:41.906327 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:41.910283 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25099
I20260806 04:33:41.910413 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:41.910666 29048 external_mini_cluster.cc:1428] Setting key a508cfeeca4bee971e435470fb5be3ae
I20260806 04:33:41.912781 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: Bootstrap starting.
I20260806 04:33:41.915563 25194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:41.916189 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:41.912140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54865 (local address 127.0.0.1:44617)
0806 04:33:41.912602 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:41.912618 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:41.912663 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:33:41.913381 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:41.913401 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:41.913713 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:41.914019 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:41.914880 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:41.914884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:41.914897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:41.914908 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:41.914964 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:41.914985 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:41.915069 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:41.915197 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 04:33:41.915355 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":124,"threads_started":1}
I20260806 04:33:41.916422 25194 log.cc:824] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:41.918776 25194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: No bootstrap required, opened a new log
I20260806 04:33:41.919085 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:41.922809 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:41.921170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34287 (local address 127.0.0.1:44617)
0806 04:33:41.921308 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:41.921313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:41.921469 (+ 156us) server_negotiation.cc:407] Connection header received
0806 04:33:41.921618 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:41.921620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:41.921653 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:41.921722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:41.922427 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:41.922430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:41.922432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:41.922442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:41.922467 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:41.922478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:41.922544 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:41.922630 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 04:33:41.922665 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:33:41.923084 25194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260806 04:33:41.923372 25194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:41.923426 25194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee72019170f41eca254edd28084f6fa, State: Initialized, Role: FOLLOWER
I20260806 04:33:41.923905 25194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260806 04:33:41.924041 25194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:41.924104 25194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:41.924206 25194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:41.925233 25194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260806 04:33:41.925578 25194 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ee72019170f41eca254edd28084f6fa; no voters:
I20260806 04:33:41.925925 25194 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:41.926040 25211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:41.926271 25211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [term 1 LEADER]: Becoming Leader. State: Replica: 7ee72019170f41eca254edd28084f6fa, State: Running, Role: LEADER
I20260806 04:33:41.926604 25211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } }
I20260806 04:33:41.927295 25194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:41.928574 25216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee72019170f41eca254edd28084f6fa. Latest consensus state: current_term: 1 leader_uuid: "7ee72019170f41eca254edd28084f6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } } }
I20260806 04:33:41.928591 25214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ee72019170f41eca254edd28084f6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee72019170f41eca254edd28084f6fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44617 } } }
I20260806 04:33:41.928764 25214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:41.929116 25216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa [sys.catalog]: This master's current role is: LEADER
W20260806 04:33:41.932300 25234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:33:41.932376 25234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:33:41.932456 25226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:41.933236 25226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:41.939165 25226 catalog_manager.cc:1347] Generated new cluster ID: dea9b0bb284b475c9e372250f3cf2fe5
I20260806 04:33:41.939260 25226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:41.947096 25226 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:41.947512 25226 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:41.954006 25226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ee72019170f41eca254edd28084f6fa: Generated new TSK 0
I20260806 04:33:41.954967 25226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:41.961651 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25099
2026-08-06T04:33:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:42 dist-test-slave-ncp9 krb5kdc[25243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:42 dist-test-slave-ncp9 krb5kdc[25243](info): set up 2 sockets
Aug 06 04:33:42 dist-test-slave-ncp9 krb5kdc[25243](info): commencing operation
krb5kdc: starting...
W20260806 04:33:43.075480 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:43.203595 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:43.319726 25268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:43.320055 25268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:43.320134 25268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:43.324317 25268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:43.324398 25268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:43.324416 25268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:43.324431 25268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:43.324451 25268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:43.329924 25268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:43.331012 25268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:43.332010 25268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:43.340556 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:43.340545 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:43.340545 25275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:43.340837 25268 server_base.cc:1034] running on GCE node
I20260806 04:33:43.341476 25268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:43.342661 25268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:43.343820 25268 hybrid_clock.cc:648] HybridClock initialized: now 1785990823343795 us; error 38 us; skew 500 ppm
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:43.346517 25268 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:43.347471 25268 webserver.cc:460] Webserver started at http://127.0.0.1:32781/ using document root <none> and password file <none>
I20260806 04:33:43.348033 25268 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:43.348101 25268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:43.348300 25268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:43.350440 25268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5f8022d302b34c07a645c6148455afd3"
format_stamp: "Formatted at 2026-08-06 04:33:43 on dist-test-slave-ncp9"
server_key: "f23b61fe24b53cc9e3a5cd2522ab6248"
server_key_iv: "a196b12b67b754bbab4f07b50b59ef0a"
server_key_version: "encryptionkey@0"
I20260806 04:33:43.351019 25268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5f8022d302b34c07a645c6148455afd3"
format_stamp: "Formatted at 2026-08-06 04:33:43 on dist-test-slave-ncp9"
server_key: "f23b61fe24b53cc9e3a5cd2522ab6248"
server_key_iv: "a196b12b67b754bbab4f07b50b59ef0a"
server_key_version: "encryptionkey@0"
I20260806 04:33:43.355012 25268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:43.357501 25290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:43.358747 25268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:43.358872 25268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5f8022d302b34c07a645c6148455afd3"
format_stamp: "Formatted at 2026-08-06 04:33:43 on dist-test-slave-ncp9"
server_key: "f23b61fe24b53cc9e3a5cd2522ab6248"
server_key_iv: "a196b12b67b754bbab4f07b50b59ef0a"
server_key_version: "encryptionkey@0"
I20260806 04:33:43.358973 25268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:43.376570 25268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:43.379377 25268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:43.379581 25268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:43.386909 25268 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46207
I20260806 04:33:43.386927 25365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46207 every 8 connection(s)
I20260806 04:33:43.387923 25268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:43.390633 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25268
I20260806 04:33:43.390779 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:43.391048 25366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:43.391105 29048 external_mini_cluster.cc:1428] Setting key d8114bd40e9f16e3c98fe70f08814862
I20260806 04:33:43.398432 25366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3: Bootstrap starting.
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:33:43.401333 25366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:43.402091 25366 log.cc:824] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:43.404522 25366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3: No bootstrap required, opened a new log
I20260806 04:33:43.405809 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:43.392500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38105 (local address 127.0.0.1:46207)
0806 04:33:43.392980 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:43.392998 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:43.393038 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:33:43.393878 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:43.393896 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:43.394259 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:43.394544 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:43.395469 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:43.396453 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:43.397634 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:43.398107 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:43.398229 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:43.401073 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:43.401078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:43.401093 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:43.401132 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:43.402960 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:43.403448 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:43.403453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:43.403458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:43.403566 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:43.403829 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:43.403831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:43.403832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:43.404233 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:43.404386 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:43.404703 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 04:33:43.405024 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260806 04:33:43.408305 25366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260806 04:33:43.408545 25366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:43.408581 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:43.408605 25366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8022d302b34c07a645c6148455afd3, State: Initialized, Role: FOLLOWER
I20260806 04:33:43.409116 25366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260806 04:33:43.409287 25366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:43.409353 25366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:43.409454 25366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:43.410691 25366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260806 04:33:43.411083 25366 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8022d302b34c07a645c6148455afd3; no voters:
I20260806 04:33:43.411459 25366 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:43.411556 25381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:43.411778 25381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [term 1 LEADER]: Becoming Leader. State: Replica: 5f8022d302b34c07a645c6148455afd3, State: Running, Role: LEADER
I20260806 04:33:43.412130 25381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260806 04:33:43.412869 25366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:43.413329 25381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8022d302b34c07a645c6148455afd3. Latest consensus state: current_term: 1 leader_uuid: "5f8022d302b34c07a645c6148455afd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } } }
I20260806 04:33:43.413455 25381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:43.414135 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8022d302b34c07a645c6148455afd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8022d302b34c07a645c6148455afd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } } }
I20260806 04:33:43.414248 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:43.415335 25391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:43.418715 25391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:43.419672 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:43.410899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40185 (local address 127.0.0.1:46207)
0806 04:33:43.411014 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:43.411018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:43.411145 (+ 127us) server_negotiation.cc:407] Connection header received
0806 04:33:43.411252 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:43.411255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:43.411305 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:43.411395 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:43.412201 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:43.413017 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:43.414216 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:43.414544 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:43.414631 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:43.416306 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:43.416310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:43.416314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:43.416340 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:43.417867 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:43.418370 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:43.418374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:43.418376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:43.418470 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:43.418761 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:43.418765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:43.418766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:43.418992 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:43.419121 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:43.419400 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 04:33:43.419524 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:33:43.424983 25391 catalog_manager.cc:1347] Generated new cluster ID: 7505bbe4906c44c4a9f00aafebfaac18
I20260806 04:33:43.425055 25391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:43.439891 25391 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:43.440856 25391 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:43.449105 25391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8022d302b34c07a645c6148455afd3: Generated new TSK 0
I20260806 04:33:43.450021 25391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:43.454519 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25268
2026-08-06T04:33:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25418](info): set up 2 sockets
Aug 06 04:33:43 dist-test-slave-ncp9 krb5kdc[25418](info): commencing operation
krb5kdc: starting...
W20260806 04:33:44.566445 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:44 dist-test-slave-ncp9 krb5kdc[25418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:44.695513 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:44.811774 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:44.812070 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:44.812129 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:44.816215 25433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:44.816294 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:44.816322 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:44.816344 25433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:44.816365 25433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:44.821832 25433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:44.822980 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:44.824049 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:44.832505 25438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:44.832546 25439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:44.832540 25442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:44.832717 25433 server_base.cc:1034] running on GCE node
I20260806 04:33:44.833396 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:44.834503 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:44.835656 25433 hybrid_clock.cc:648] HybridClock initialized: now 1785990824835641 us; error 27 us; skew 500 ppm
Aug 06 04:33:44 dist-test-slave-ncp9 krb5kdc[25418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:44.838378 25433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:44.839301 25433 webserver.cc:460] Webserver started at http://127.0.0.1:35791/ using document root <none> and password file <none>
I20260806 04:33:44.839851 25433 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:44.839912 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:44.840124 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:44.842273 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "6e03ffc947de485f8ee254aace05f82d"
format_stamp: "Formatted at 2026-08-06 04:33:44 on dist-test-slave-ncp9"
server_key: "3a205f691cae54eb9d24e4eead2e9d84"
server_key_iv: "36096fab6e9f6a2e2cceddc4bae7d1c3"
server_key_version: "encryptionkey@0"
I20260806 04:33:44.842836 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "6e03ffc947de485f8ee254aace05f82d"
format_stamp: "Formatted at 2026-08-06 04:33:44 on dist-test-slave-ncp9"
server_key: "3a205f691cae54eb9d24e4eead2e9d84"
server_key_iv: "36096fab6e9f6a2e2cceddc4bae7d1c3"
server_key_version: "encryptionkey@0"
I20260806 04:33:44.846611 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:33:44.849028 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:44.850237 25433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:33:44.850348 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "6e03ffc947de485f8ee254aace05f82d"
format_stamp: "Formatted at 2026-08-06 04:33:44 on dist-test-slave-ncp9"
server_key: "3a205f691cae54eb9d24e4eead2e9d84"
server_key_iv: "36096fab6e9f6a2e2cceddc4bae7d1c3"
server_key_version: "encryptionkey@0"
I20260806 04:33:44.850445 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:44.872324 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:44.875659 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:44.875861 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:44.883159 25433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36607
I20260806 04:33:44.883210 25539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36607 every 8 connection(s)
I20260806 04:33:44.884173 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:44.887071 25541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:44.893138 25541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d: Bootstrap starting.
I20260806 04:33:44.893576 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25433
I20260806 04:33:44.893704 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:44.893936 29048 external_mini_cluster.cc:1428] Setting key 100a754336847ec1b70ecec48704b7ae
I20260806 04:33:44.895833 25541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:44.896600 25541 log.cc:824] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:44.898761 25541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d: No bootstrap required, opened a new log
I20260806 04:33:44.903192 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:44.895331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60731 (local address 127.0.0.1:36607)
0806 04:33:44.895758 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:44.895775 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:44.895811 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:33:44.896466 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:44.896483 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:44.896846 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:44.897176 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:44.897641 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:44.898858 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:44.900046 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:44.900504 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:44.900614 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:44.901487 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:44.901492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:44.901506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:44.901892 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:44.901945 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:44.901969 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:44.902042 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:44.902206 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 04:33:44.902403 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260806 04:33:44.903047 25541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260806 04:33:44.903273 25541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:44.903318 25541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e03ffc947de485f8ee254aace05f82d, State: Initialized, Role: FOLLOWER
I20260806 04:33:44.903762 25541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260806 04:33:44.903901 25541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:44.903983 25541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:44.904101 25541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:44.905196 25541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260806 04:33:44.905592 25541 leader_election.cc:302] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e03ffc947de485f8ee254aace05f82d; no voters:
I20260806 04:33:44.906015 25541 leader_election.cc:288] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:44.906093 25549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:44.906176 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:44.906306 25549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [term 1 LEADER]: Becoming Leader. State: Replica: 6e03ffc947de485f8ee254aace05f82d, State: Running, Role: LEADER
I20260806 04:33:44.906641 25549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20260806 04:33:44.907763 25541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:44.907783 25549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e03ffc947de485f8ee254aace05f82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20260806 04:33:44.907909 25549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:44.908128 25549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e03ffc947de485f8ee254aace05f82d. Latest consensus state: current_term: 1 leader_uuid: "6e03ffc947de485f8ee254aace05f82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03ffc947de485f8ee254aace05f82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20260806 04:33:44.908214 25549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:44.908463 25559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:44.911911 25559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:44.915053 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:44.908830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58201 (local address 127.0.0.1:36607)
0806 04:33:44.908960 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:44.908965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:44.910357 (+ 1392us) server_negotiation.cc:407] Connection header received
0806 04:33:44.910565 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:44.910569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:44.910617 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:44.910710 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:44.911152 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:44.911920 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:44.913261 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:44.913580 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:44.913697 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:44.914610 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:44.914614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:44.914617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:44.914630 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:44.914661 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:44.914674 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:44.914743 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:44.914861 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 04:33:44.914916 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 04:33:44.917584 25559 catalog_manager.cc:1347] Generated new cluster ID: ea8d6eefbcda490db8fd26ced0817974
I20260806 04:33:44.917660 25559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:44.926872 25559 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:44.928238 25559 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:44.934286 25559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e03ffc947de485f8ee254aace05f82d: Generated new TSK 0
I20260806 04:33:44.934958 25559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:44.945856 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25433
2026-08-06T04:33:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:45 dist-test-slave-ncp9 krb5kdc[25588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:45 dist-test-slave-ncp9 krb5kdc[25588](info): set up 2 sockets
Aug 06 04:33:45 dist-test-slave-ncp9 krb5kdc[25588](info): commencing operation
krb5kdc: starting...
W20260806 04:33:46.063532 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:46 dist-test-slave-ncp9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:46.188397 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:46.301964 25611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:46.302259 25611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:46.302301 25611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:46.306239 25611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:46.306298 25611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:46.306316 25611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:46.306330 25611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:46.306344 25611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:46.311585 25611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:46.312649 25611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:46.313701 25611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:46.322016 25617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:46.322065 25622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:46.322111 25611 server_base.cc:1034] running on GCE node
W20260806 04:33:46.322010 25616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:46.322826 25611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:46.323925 25611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:46.325084 25611 hybrid_clock.cc:648] HybridClock initialized: now 1785990826325067 us; error 33 us; skew 500 ppm
Aug 06 04:33:46 dist-test-slave-ncp9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:46.327723 25611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:46.328601 25611 webserver.cc:460] Webserver started at http://127.0.0.1:41495/ using document root <none> and password file <none>
I20260806 04:33:46.329190 25611 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:46.329252 25611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:46.329435 25611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:46.331434 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "f684b52e26c545958bd983f183e790c7"
format_stamp: "Formatted at 2026-08-06 04:33:46 on dist-test-slave-ncp9"
server_key: "ee621b6bcc1bba3bc8474fd4c3831b48"
server_key_iv: "6cb0bf613df2190770203444986f1b56"
server_key_version: "encryptionkey@0"
I20260806 04:33:46.331938 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "f684b52e26c545958bd983f183e790c7"
format_stamp: "Formatted at 2026-08-06 04:33:46 on dist-test-slave-ncp9"
server_key: "ee621b6bcc1bba3bc8474fd4c3831b48"
server_key_iv: "6cb0bf613df2190770203444986f1b56"
server_key_version: "encryptionkey@0"
I20260806 04:33:46.335528 25611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:33:46.337872 25634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:46.339067 25611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:46.339176 25611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "f684b52e26c545958bd983f183e790c7"
format_stamp: "Formatted at 2026-08-06 04:33:46 on dist-test-slave-ncp9"
server_key: "ee621b6bcc1bba3bc8474fd4c3831b48"
server_key_iv: "6cb0bf613df2190770203444986f1b56"
server_key_version: "encryptionkey@0"
I20260806 04:33:46.339277 25611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:46.343868 25611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:46.346664 25611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:46.346825 25611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:46.353762 25611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42217
I20260806 04:33:46.353777 25706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42217 every 8 connection(s)
I20260806 04:33:46.354735 25611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:46.357570 25707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:46.363176 25707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7: Bootstrap starting.
I20260806 04:33:46.364804 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25611
I20260806 04:33:46.364962 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:46.365270 29048 external_mini_cluster.cc:1428] Setting key c4483141e6319011e26d65fee9a93162
I20260806 04:33:46.365926 25707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:46.366829 25707 log.cc:824] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:46.368775 25707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7: No bootstrap required, opened a new log
I20260806 04:33:46.370306 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:46.366412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46417 (local address 127.0.0.1:42217)
0806 04:33:46.366856 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:46.366870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:46.366906 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:33:46.367723 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:46.367744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:46.368019 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:46.368329 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:46.369147 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:46.369152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:46.369164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:46.369174 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:46.369227 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:46.369246 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:46.369341 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:46.369467 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 04:33:46.369631 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260806 04:33:46.372692 25707 raft_consensus.cc:348] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260806 04:33:46.372991 25707 raft_consensus.cc:374] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:46.373054 25707 raft_consensus.cc:729] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f684b52e26c545958bd983f183e790c7, State: Initialized, Role: FOLLOWER
I20260806 04:33:46.373241 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:46.373485 25707 consensus_queue.cc:260] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260806 04:33:46.373623 25707 raft_consensus.cc:388] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:46.373687 25707 raft_consensus.cc:482] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:46.373780 25707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:46.374709 25707 raft_consensus.cc:504] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260806 04:33:46.375044 25707 leader_election.cc:302] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f684b52e26c545958bd983f183e790c7; no voters:
I20260806 04:33:46.375361 25707 leader_election.cc:288] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:46.375707 25720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:46.375905 25720 raft_consensus.cc:686] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [term 1 LEADER]: Becoming Leader. State: Replica: f684b52e26c545958bd983f183e790c7, State: Running, Role: LEADER
I20260806 04:33:46.376240 25720 consensus_queue.cc:237] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } }
I20260806 04:33:46.376757 25707 sys_catalog.cc:564] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:46.377260 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:46.375234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45327 (local address 127.0.0.1:42217)
0806 04:33:46.375383 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:46.375387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:46.375500 (+ 113us) server_negotiation.cc:407] Connection header received
0806 04:33:46.375601 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:46.375604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:46.375639 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:46.376507 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:46.376594 (+ 87us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:46.376597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:46.376599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:46.376607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:46.376624 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:46.376634 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:46.377078 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:46.377113 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 04:33:46.377150 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:33:46.377748 25722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f684b52e26c545958bd983f183e790c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260806 04:33:46.377868 25722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:46.378084 25722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f684b52e26c545958bd983f183e790c7. Latest consensus state: current_term: 1 leader_uuid: "f684b52e26c545958bd983f183e790c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f684b52e26c545958bd983f183e790c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42217 } } }
I20260806 04:33:46.378158 25722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:46.378517 25732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:46.382084 25732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:46.387301 25732 catalog_manager.cc:1347] Generated new cluster ID: 47478e5778ba445ab1e56a000d828459
I20260806 04:33:46.387355 25732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:46.415244 25732 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:46.415565 25732 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:46.420877 25732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f684b52e26c545958bd983f183e790c7: Generated new TSK 0
I20260806 04:33:46.421567 25732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:46.435738 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25611
2026-08-06T04:33:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:46 dist-test-slave-ncp9 krb5kdc[25746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:46 dist-test-slave-ncp9 krb5kdc[25746](info): set up 2 sockets
Aug 06 04:33:46 dist-test-slave-ncp9 krb5kdc[25746](info): commencing operation
krb5kdc: starting...
W20260806 04:33:47.546104 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:47.670135 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:47.787016 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:47.787289 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:47.787353 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:47.791339 25767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:47.791414 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:47.791455 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:47.791482 25767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:47.791515 25767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:47.796725 25767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:47.797829 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:47.799077 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:47.807615 25776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:47.807639 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:47.807770 25767 server_base.cc:1034] running on GCE node
W20260806 04:33:47.807673 25774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:47.808501 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:47.809628 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:47.810803 25767 hybrid_clock.cc:648] HybridClock initialized: now 1785990827810781 us; error 31 us; skew 500 ppm
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:47.817685 25767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:47.818629 25767 webserver.cc:460] Webserver started at http://127.0.0.1:45033/ using document root <none> and password file <none>
I20260806 04:33:47.819186 25767 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:47.819255 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:47.819468 25767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:47.821602 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "d982804051044bee9e703f7d0b60e7d4"
format_stamp: "Formatted at 2026-08-06 04:33:47 on dist-test-slave-ncp9"
server_key: "390a85602d85c644e538bac8e39e5b4b"
server_key_iv: "3d1fd85274a965885b48b9c26dbb1a3b"
server_key_version: "encryptionkey@0"
I20260806 04:33:47.822136 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "d982804051044bee9e703f7d0b60e7d4"
format_stamp: "Formatted at 2026-08-06 04:33:47 on dist-test-slave-ncp9"
server_key: "390a85602d85c644e538bac8e39e5b4b"
server_key_iv: "3d1fd85274a965885b48b9c26dbb1a3b"
server_key_version: "encryptionkey@0"
I20260806 04:33:47.825846 25767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:47.828289 25788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:47.829584 25767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:47.829713 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "d982804051044bee9e703f7d0b60e7d4"
format_stamp: "Formatted at 2026-08-06 04:33:47 on dist-test-slave-ncp9"
server_key: "390a85602d85c644e538bac8e39e5b4b"
server_key_iv: "3d1fd85274a965885b48b9c26dbb1a3b"
server_key_version: "encryptionkey@0"
I20260806 04:33:47.829818 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:47.846261 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:47.849637 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:47.849840 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:47.857254 25767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36513
I20260806 04:33:47.857261 25865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36513 every 8 connection(s)
I20260806 04:33:47.858278 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:47.861198 25866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:47.866767 25866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4: Bootstrap starting.
I20260806 04:33:47.867726 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25767
I20260806 04:33:47.867859 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:47.868161 29048 external_mini_cluster.cc:1428] Setting key 1320af4a07afec6ecf1290e2c9b47161
I20260806 04:33:47.869706 25866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:47.870430 25866 log.cc:824] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:47.872598 25866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4: No bootstrap required, opened a new log
I20260806 04:33:47.876803 25866 raft_consensus.cc:348] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260806 04:33:47.877074 25866 raft_consensus.cc:374] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:47.877135 25866 raft_consensus.cc:729] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d982804051044bee9e703f7d0b60e7d4, State: Initialized, Role: FOLLOWER
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:33:47.877574 25866 consensus_queue.cc:260] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260806 04:33:47.877708 25866 raft_consensus.cc:388] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:47.877775 25866 raft_consensus.cc:482] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:47.877878 25866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:47.879112 25866 raft_consensus.cc:504] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260806 04:33:47.879504 25866 leader_election.cc:302] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d982804051044bee9e703f7d0b60e7d4; no voters:
I20260806 04:33:47.879889 25866 leader_election.cc:288] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:47.879978 25874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:47.880194 25874 raft_consensus.cc:686] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [term 1 LEADER]: Becoming Leader. State: Replica: d982804051044bee9e703f7d0b60e7d4, State: Running, Role: LEADER
I20260806 04:33:47.880589 25874 consensus_queue.cc:237] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } }
I20260806 04:33:47.881282 25866 sys_catalog.cc:564] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:47.882514 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d982804051044bee9e703f7d0b60e7d4. Latest consensus state: current_term: 1 leader_uuid: "d982804051044bee9e703f7d0b60e7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260806 04:33:47.882711 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:47.883143 25890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:47.883734 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d982804051044bee9e703f7d0b60e7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d982804051044bee9e703f7d0b60e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36513 } } }
I20260806 04:33:47.883832 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:47.883855 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:47.869398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:44215 (local address 127.0.0.1:36513)
0806 04:33:47.869834 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:47.869851 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:47.869889 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:33:47.870540 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:47.870564 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:47.870996 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:47.871352 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:47.872195 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:47.873268 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:47.874426 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:47.874910 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:47.875047 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:47.877905 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:47.877911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:47.877927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:47.877964 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:47.879863 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:47.880484 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:47.880491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:47.880497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:47.880653 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:47.881072 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:47.881075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:47.881077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:47.881660 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:47.882447 (+ 787us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:47.882639 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 04:33:47.882880 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20260806 04:33:47.886361 25890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:47.887308 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:47.892280 25890 catalog_manager.cc:1347] Generated new cluster ID: 80f5886b06de44b0bcbc7b3a9b82f11a
I20260806 04:33:47.892340 25890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:47.898113 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:47.889489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:38513 (local address 127.0.0.1:36513)
0806 04:33:47.889594 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:47.889598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:47.889829 (+ 231us) server_negotiation.cc:407] Connection header received
0806 04:33:47.889940 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:47.889944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:47.889995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:47.890051 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:47.890736 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:47.891550 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:47.892703 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:47.893073 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:47.893175 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:47.894792 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:47.894795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:47.894798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:47.894822 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:47.896387 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:47.896924 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:47.896927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:47.896929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:47.897003 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:47.897314 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:47.897316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:47.897318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:47.897507 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:47.897628 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:47.897875 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 04:33:47.897965 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:33:47.904156 25890 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:47.905071 25890 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:47.911142 25890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d982804051044bee9e703f7d0b60e7d4: Generated new TSK 0
I20260806 04:33:47.911827 25890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:47.916841 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25767
W20260806 04:33:47.925969 25744 connection.cc:535] client connection to 127.0.0.1:36513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36513): Connection reset by peer (error 104)
2026-08-06T04:33:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25910](info): set up 2 sockets
Aug 06 04:33:47 dist-test-slave-ncp9 krb5kdc[25910](info): commencing operation
krb5kdc: starting...
W20260806 04:33:49.014580 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:49 dist-test-slave-ncp9 krb5kdc[25910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:49.145517 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:49.260545 25934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:49.260842 25934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:49.260923 25934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:49.265098 25934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:49.265175 25934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:49.265202 25934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:49.265225 25934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:49.265249 25934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:49.270737 25934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:49.271894 25934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:49.272980 25934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:49.281417 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:49.281430 25941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:49.281560 25934 server_base.cc:1034] running on GCE node
W20260806 04:33:49.281431 25944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:49.282307 25934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:49.283411 25934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:49.284577 25934 hybrid_clock.cc:648] HybridClock initialized: now 1785990829284564 us; error 28 us; skew 500 ppm
Aug 06 04:33:49 dist-test-slave-ncp9 krb5kdc[25910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:49.287353 25934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:49.288280 25934 webserver.cc:460] Webserver started at http://127.0.0.1:46315/ using document root <none> and password file <none>
I20260806 04:33:49.288832 25934 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:49.288923 25934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:49.289155 25934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:49.291215 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "32e95df4d7194cb99ab321e7c36244fc"
format_stamp: "Formatted at 2026-08-06 04:33:49 on dist-test-slave-ncp9"
server_key: "4c1e82985c9808eaab7b1a4f50ac421d"
server_key_iv: "07e3597084710876eee8ebf12dcd3aff"
server_key_version: "encryptionkey@0"
I20260806 04:33:49.291761 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "32e95df4d7194cb99ab321e7c36244fc"
format_stamp: "Formatted at 2026-08-06 04:33:49 on dist-test-slave-ncp9"
server_key: "4c1e82985c9808eaab7b1a4f50ac421d"
server_key_iv: "07e3597084710876eee8ebf12dcd3aff"
server_key_version: "encryptionkey@0"
I20260806 04:33:49.295578 25934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:33:49.298065 25954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:49.299342 25934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:33:49.299463 25934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "32e95df4d7194cb99ab321e7c36244fc"
format_stamp: "Formatted at 2026-08-06 04:33:49 on dist-test-slave-ncp9"
server_key: "4c1e82985c9808eaab7b1a4f50ac421d"
server_key_iv: "07e3597084710876eee8ebf12dcd3aff"
server_key_version: "encryptionkey@0"
I20260806 04:33:49.299563 25934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:49.338266 25934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:49.341218 25934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:49.341403 25934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:49.348313 25934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40125
I20260806 04:33:49.348347 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40125 every 8 connection(s)
I20260806 04:33:49.349298 25934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:49.352114 26030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:49.353330 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 25934
I20260806 04:33:49.353466 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:49.353771 29048 external_mini_cluster.cc:1428] Setting key 6634a8b276b222c0815130657a866837
I20260806 04:33:49.358405 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc: Bootstrap starting.
I20260806 04:33:49.361115 26030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:49.361961 26030 log.cc:824] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:49.362749 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:49.355119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:51419 (local address 127.0.0.1:40125)
0806 04:33:49.355524 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:49.355540 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:49.355577 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:33:49.356204 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:49.356221 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:49.356559 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:49.356826 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:49.357273 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:49.358227 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:49.359391 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:49.359859 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:49.359984 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:49.360858 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:49.360864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:49.360877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:49.361330 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:49.361387 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:49.361411 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:49.361513 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:49.361687 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:33:49.361819 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260806 04:33:49.364037 26030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc: No bootstrap required, opened a new log
I20260806 04:33:49.365661 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:49.368319 26030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } }
I20260806 04:33:49.368642 26030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:49.368706 26030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e95df4d7194cb99ab321e7c36244fc, State: Initialized, Role: FOLLOWER
I20260806 04:33:49.369169 26030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } }
I20260806 04:33:49.369307 26030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:49.369374 26030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:49.369472 26030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:49.370389 26030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } }
I20260806 04:33:49.370730 26030 leader_election.cc:302] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e95df4d7194cb99ab321e7c36244fc; no voters:
I20260806 04:33:49.371090 26030 leader_election.cc:288] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:49.371191 26046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:49.371472 26046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [term 1 LEADER]: Becoming Leader. State: Replica: 32e95df4d7194cb99ab321e7c36244fc, State: Running, Role: LEADER
I20260806 04:33:49.372016 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:49.368009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:58243 (local address 127.0.0.1:40125)
0806 04:33:49.368136 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:49.368140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:49.368315 (+ 175us) server_negotiation.cc:407] Connection header received
0806 04:33:49.368402 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:49.368405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:49.368452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:49.368528 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:49.368779 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:49.369416 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:49.370267 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:49.370526 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:49.370619 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:49.371491 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:49.371495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:49.371498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:49.371511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:49.371539 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:49.371552 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:49.371636 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:49.371849 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 04:33:49.371890 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:33:49.371793 26046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } }
I20260806 04:33:49.372558 26030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:49.373561 26046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e95df4d7194cb99ab321e7c36244fc. Latest consensus state: current_term: 1 leader_uuid: "32e95df4d7194cb99ab321e7c36244fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } } }
I20260806 04:33:49.373680 26046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:49.374014 26051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e95df4d7194cb99ab321e7c36244fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e95df4d7194cb99ab321e7c36244fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40125 } } }
I20260806 04:33:49.374162 26051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:49.374189 26064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:49.377549 26064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:49.383623 26064 catalog_manager.cc:1347] Generated new cluster ID: 0f57e40d8d4142f293a279dbcc8e3236
I20260806 04:33:49.383705 26064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:49.401198 26064 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:49.402182 26064 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:49.413935 26064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e95df4d7194cb99ab321e7c36244fc: Generated new TSK 0
I20260806 04:33:49.414660 26064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:49.421588 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 25934
W20260806 04:33:49.433193 25908 connection.cc:535] client connection to 127.0.0.1:40125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40125): Connection reset by peer (error 104)
2026-08-06T04:33:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:49 dist-test-slave-ncp9 krb5kdc[26086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:49 dist-test-slave-ncp9 krb5kdc[26086](info): set up 2 sockets
Aug 06 04:33:49 dist-test-slave-ncp9 krb5kdc[26086](info): commencing operation
krb5kdc: starting...
W20260806 04:33:50.531629 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:50 dist-test-slave-ncp9 krb5kdc[26086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:50.659679 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:50.776718 26101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:50.777036 26101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:50.777099 26101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:50.781034 26101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:50.781100 26101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:50.781127 26101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:50.781148 26101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:50.781176 26101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:50.786417 26101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:50.787467 26101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:50.788491 26101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:50.796671 26112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:50.796842 26101 server_base.cc:1034] running on GCE node
W20260806 04:33:50.796622 26109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:50.796645 26110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:50.797581 26101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:50.798638 26101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:50.799793 26101 hybrid_clock.cc:648] HybridClock initialized: now 1785990830799777 us; error 27 us; skew 500 ppm
Aug 06 04:33:50 dist-test-slave-ncp9 krb5kdc[26086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:50.802484 26101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:50.803390 26101 webserver.cc:460] Webserver started at http://127.0.0.1:44629/ using document root <none> and password file <none>
I20260806 04:33:50.803911 26101 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:50.803968 26101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:50.804142 26101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:50.806154 26101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "b5168259f21947ee80b225402b849e43"
format_stamp: "Formatted at 2026-08-06 04:33:50 on dist-test-slave-ncp9"
server_key: "102524f1559a0dfa75c1c7ede018f422"
server_key_iv: "cd511e958bdfe7e2f496c82ecbaf57da"
server_key_version: "encryptionkey@0"
I20260806 04:33:50.806659 26101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "b5168259f21947ee80b225402b849e43"
format_stamp: "Formatted at 2026-08-06 04:33:50 on dist-test-slave-ncp9"
server_key: "102524f1559a0dfa75c1c7ede018f422"
server_key_iv: "cd511e958bdfe7e2f496c82ecbaf57da"
server_key_version: "encryptionkey@0"
I20260806 04:33:50.810271 26101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:33:50.812558 26123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:50.813757 26101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:50.813871 26101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "b5168259f21947ee80b225402b849e43"
format_stamp: "Formatted at 2026-08-06 04:33:50 on dist-test-slave-ncp9"
server_key: "102524f1559a0dfa75c1c7ede018f422"
server_key_iv: "cd511e958bdfe7e2f496c82ecbaf57da"
server_key_version: "encryptionkey@0"
I20260806 04:33:50.813961 26101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:50.818338 26101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:50.821013 26101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:50.821171 26101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:50.827925 26101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34959
I20260806 04:33:50.827932 26196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34959 every 8 connection(s)
I20260806 04:33:50.828812 26101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:50.831568 26197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:50.836082 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26101
I20260806 04:33:50.836205 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:50.836468 29048 external_mini_cluster.cc:1428] Setting key 3a0f0edb7fb027d05febedc7ca32de08
I20260806 04:33:50.837091 26197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43: Bootstrap starting.
I20260806 04:33:50.839712 26197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:50.840607 26197 log.cc:824] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:50.841774 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:50.837847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:52479 (local address 127.0.0.1:34959)
0806 04:33:50.838283 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:50.838301 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:50.838345 (+ 44us) server_negotiation.cc:407] Connection header received
0806 04:33:50.839136 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:50.839160 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:50.839501 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:50.839868 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:50.840478 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:50.840483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:50.840497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:50.840509 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:50.840564 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:50.840588 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:50.840717 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:50.840798 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 04:33:50.840986 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260806 04:33:50.842612 26197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43: No bootstrap required, opened a new log
I20260806 04:33:50.844426 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:50.846479 26197 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260806 04:33:50.846731 26197 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:50.846796 26197 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5168259f21947ee80b225402b849e43, State: Initialized, Role: FOLLOWER
I20260806 04:33:50.847255 26197 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260806 04:33:50.847411 26197 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:50.847481 26197 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:50.847585 26197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:50.847784 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:50.846378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:39953 (local address 127.0.0.1:34959)
0806 04:33:50.846482 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:50.846485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:50.846593 (+ 108us) server_negotiation.cc:407] Connection header received
0806 04:33:50.846673 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:50.846675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:50.846709 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:50.846789 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:50.847395 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:50.847398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:50.847400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:50.847413 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:50.847442 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:50.847456 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:50.847541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:50.847633 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 04:33:50.847682 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:33:50.848541 26197 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260806 04:33:50.848867 26197 leader_election.cc:302] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5168259f21947ee80b225402b849e43; no voters:
I20260806 04:33:50.849192 26197 leader_election.cc:288] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:50.849277 26209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:50.849458 26209 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [term 1 LEADER]: Becoming Leader. State: Replica: b5168259f21947ee80b225402b849e43, State: Running, Role: LEADER
I20260806 04:33:50.849802 26209 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260806 04:33:50.850293 26197 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:50.851658 26211 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5168259f21947ee80b225402b849e43. Latest consensus state: current_term: 1 leader_uuid: "b5168259f21947ee80b225402b849e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } } }
I20260806 04:33:50.851771 26211 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:50.852507 26210 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5168259f21947ee80b225402b849e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5168259f21947ee80b225402b849e43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } } }
I20260806 04:33:50.852609 26210 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:50.852614 26226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:50.855155 26226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:50.859938 26226 catalog_manager.cc:1347] Generated new cluster ID: 63c10a46dda74e50a2f023a548acca53
I20260806 04:33:50.859997 26226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:50.870450 26226 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:50.870746 26226 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:50.878765 26226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5168259f21947ee80b225402b849e43: Generated new TSK 0
I20260806 04:33:50.879323 26226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:50.882779 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26101
2026-08-06T04:33:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:50 dist-test-slave-ncp9 krb5kdc[26239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:50 dist-test-slave-ncp9 krb5kdc[26239](info): set up 2 sockets
Aug 06 04:33:50 dist-test-slave-ncp9 krb5kdc[26239](info): commencing operation
krb5kdc: starting...
W20260806 04:33:51.977909 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:52.101677 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:52.214349 26258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:52.214627 26258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:52.214670 26258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:52.218595 26258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:52.218650 26258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:52.218669 26258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:52.218684 26258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:52.218698 26258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:52.224010 26258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:52.225107 26258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:52.226117 26258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:52.234717 26265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:52.234823 26258 server_base.cc:1034] running on GCE node
W20260806 04:33:52.234717 26268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:52.234716 26266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:52.235533 26258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:52.236639 26258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:52.237836 26258 hybrid_clock.cc:648] HybridClock initialized: now 1785990832237814 us; error 32 us; skew 500 ppm
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:52.240545 26258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:52.241490 26258 webserver.cc:460] Webserver started at http://127.0.0.1:43725/ using document root <none> and password file <none>
I20260806 04:33:52.242038 26258 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:52.242105 26258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:52.242340 26258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:52.244371 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "5df21a17ce19408180b93d80e417122f"
format_stamp: "Formatted at 2026-08-06 04:33:52 on dist-test-slave-ncp9"
server_key: "35b5a79c8084a7a74082d65cc0ea9ae7"
server_key_iv: "2818afded1a81c0a83b1ddab11c0f7b9"
server_key_version: "encryptionkey@0"
I20260806 04:33:52.244957 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "5df21a17ce19408180b93d80e417122f"
format_stamp: "Formatted at 2026-08-06 04:33:52 on dist-test-slave-ncp9"
server_key: "35b5a79c8084a7a74082d65cc0ea9ae7"
server_key_iv: "2818afded1a81c0a83b1ddab11c0f7b9"
server_key_version: "encryptionkey@0"
I20260806 04:33:52.248571 26258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:52.250964 26279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:52.252192 26258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:33:52.252310 26258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "5df21a17ce19408180b93d80e417122f"
format_stamp: "Formatted at 2026-08-06 04:33:52 on dist-test-slave-ncp9"
server_key: "35b5a79c8084a7a74082d65cc0ea9ae7"
server_key_iv: "2818afded1a81c0a83b1ddab11c0f7b9"
server_key_version: "encryptionkey@0"
I20260806 04:33:52.252416 26258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:52.273877 26258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:52.277038 26258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:52.277225 26258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:52.284277 26258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35245
I20260806 04:33:52.284291 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35245 every 8 connection(s)
I20260806 04:33:52.285259 26258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:52.288187 26360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:52.288326 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26258
I20260806 04:33:52.288445 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:52.288769 29048 external_mini_cluster.cc:1428] Setting key 1f9f8db6aaae8d8d6aa8fc76eac0b0cd
I20260806 04:33:52.294633 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f: Bootstrap starting.
I20260806 04:33:52.296942 26360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f: Neither blocks nor log segments found. Creating new log.
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:33:52.297783 26360 log.cc:824] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:52.299638 26360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f: No bootstrap required, opened a new log
I20260806 04:33:52.302299 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:52.290287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48133 (local address 127.0.0.1:35245)
0806 04:33:52.290726 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:52.290740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:52.290777 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:33:52.291464 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:52.291484 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:52.291817 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:52.292130 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:52.292923 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:52.293896 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:52.294726 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:52.295208 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:52.295360 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:52.297928 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:52.297935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:52.297952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:52.297990 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:52.299504 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:52.300010 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:52.300016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:52.300021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:52.300128 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:52.300433 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:52.300436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:52.300437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:52.300836 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:52.301059 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:52.301371 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 04:33:52.301580 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260806 04:33:52.303362 26360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } }
I20260806 04:33:52.303645 26360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:52.303725 26360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df21a17ce19408180b93d80e417122f, State: Initialized, Role: FOLLOWER
I20260806 04:33:52.304167 26360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } }
I20260806 04:33:52.304323 26360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:52.304388 26360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:52.304486 26360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:52.305336 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:52.305604 26360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } }
I20260806 04:33:52.305920 26360 leader_election.cc:302] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df21a17ce19408180b93d80e417122f; no voters:
I20260806 04:33:52.306212 26360 leader_election.cc:288] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:52.306306 26365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:52.306501 26365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [term 1 LEADER]: Becoming Leader. State: Replica: 5df21a17ce19408180b93d80e417122f, State: Running, Role: LEADER
I20260806 04:33:52.306855 26365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } }
I20260806 04:33:52.307569 26360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:52.308091 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df21a17ce19408180b93d80e417122f. Latest consensus state: current_term: 1 leader_uuid: "5df21a17ce19408180b93d80e417122f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } } }
I20260806 04:33:52.308260 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:52.308780 26368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df21a17ce19408180b93d80e417122f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df21a17ce19408180b93d80e417122f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35245 } } }
I20260806 04:33:52.308876 26368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:52.309024 26383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:52.311877 26383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:52.318288 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:52.307519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48981 (local address 127.0.0.1:35245)
0806 04:33:52.307650 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:52.307655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:52.307920 (+ 265us) server_negotiation.cc:407] Connection header received
0806 04:33:52.308987 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:52.308991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:52.309047 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:52.309105 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:52.310145 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:52.310959 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:52.312183 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:52.312521 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:52.312598 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:52.314281 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:52.314285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:52.314288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:52.314315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:52.315998 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:52.316667 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:52.316671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:52.316673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:52.316763 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:52.317225 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:52.317229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:52.317231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:52.317485 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:52.317595 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:52.318011 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 04:33:52.318113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:33:52.318663 26383 catalog_manager.cc:1347] Generated new cluster ID: c9c5b055852c4ad7884455bd1c6a71c5
I20260806 04:33:52.318727 26383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:52.351171 26383 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:52.352489 26383 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:52.360508 26383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df21a17ce19408180b93d80e417122f: Generated new TSK 0
I20260806 04:33:52.361400 26383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:52.373584 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26258
W20260806 04:33:52.387888 26236 connection.cc:535] client connection to 127.0.0.1:35245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35245): Connection reset by peer (error 104)
2026-08-06T04:33:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26403](info): set up 2 sockets
Aug 06 04:33:52 dist-test-slave-ncp9 krb5kdc[26403](info): commencing operation
krb5kdc: starting...
W20260806 04:33:53.481704 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:53 dist-test-slave-ncp9 krb5kdc[26403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:53.608122 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:53.721565 26417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:53.721864 26417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:53.721925 26417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:53.726008 26417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:53.726078 26417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:53.726105 26417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:53.726128 26417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:53.726157 26417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:53.731578 26417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:53.732645 26417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:53.733700 26417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:53.741947 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:53.741962 26427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:53.742035 26428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:53.742223 26417 server_base.cc:1034] running on GCE node
I20260806 04:33:53.742827 26417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:53.743873 26417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:53.745040 26417 hybrid_clock.cc:648] HybridClock initialized: now 1785990833745003 us; error 51 us; skew 500 ppm
Aug 06 04:33:53 dist-test-slave-ncp9 krb5kdc[26403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:53.750929 26417 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:53.751813 26417 webserver.cc:460] Webserver started at http://127.0.0.1:39613/ using document root <none> and password file <none>
I20260806 04:33:53.752354 26417 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:53.752419 26417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:53.752632 26417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:53.754650 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3"
format_stamp: "Formatted at 2026-08-06 04:33:53 on dist-test-slave-ncp9"
server_key: "f2667b487ccb00023c46d4619f793a50"
server_key_iv: "b7678b4f6b76d08d425c8258b4a01410"
server_key_version: "encryptionkey@0"
I20260806 04:33:53.755172 26417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3"
format_stamp: "Formatted at 2026-08-06 04:33:53 on dist-test-slave-ncp9"
server_key: "f2667b487ccb00023c46d4619f793a50"
server_key_iv: "b7678b4f6b76d08d425c8258b4a01410"
server_key_version: "encryptionkey@0"
I20260806 04:33:53.758761 26417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 04:33:53.761178 26441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:53.762427 26417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:33:53.762550 26417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3"
format_stamp: "Formatted at 2026-08-06 04:33:53 on dist-test-slave-ncp9"
server_key: "f2667b487ccb00023c46d4619f793a50"
server_key_iv: "b7678b4f6b76d08d425c8258b4a01410"
server_key_version: "encryptionkey@0"
I20260806 04:33:53.762645 26417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:53.781697 26417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:53.784783 26417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:53.785006 26417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:53.791914 26417 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43225
I20260806 04:33:53.791929 26518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43225 every 8 connection(s)
I20260806 04:33:53.792881 26417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:53.795079 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26417
I20260806 04:33:53.795217 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:53.795517 29048 external_mini_cluster.cc:1428] Setting key d84c516256e12a28166cfe4bb553107a
I20260806 04:33:53.796036 26519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:53.802112 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3: Bootstrap starting.
I20260806 04:33:53.804296 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:53.796988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:53655 (local address 127.0.0.1:43225)
0806 04:33:53.797442 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:53.797458 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:53.797495 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:33:53.798150 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:53.798167 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:53.798496 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:53.798794 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:53.799204 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:53.800182 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:53.801089 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:53.801594 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:53.801738 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:53.802619 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:53.802626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:53.802643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:53.803043 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:53.803098 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:53.803122 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:53.803237 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:53.803420 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 04:33:53.803558 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260806 04:33:53.804442 26519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:53.805262 26519 log.cc:824] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:53.807228 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:53.807418 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3: No bootstrap required, opened a new log
I20260806 04:33:53.811161 26519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } }
I20260806 04:33:53.811376 26519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:53.811460 26519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aeb0cea9f24d47a5a26fe3f8c6d7e3, State: Initialized, Role: FOLLOWER
I20260806 04:33:53.811937 26519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } }
I20260806 04:33:53.812107 26519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:53.812182 26519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:53.812289 26519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:53.812696 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:53.809111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:48331 (local address 127.0.0.1:43225)
0806 04:33:53.809213 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:53.809217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:53.809411 (+ 194us) server_negotiation.cc:407] Connection header received
0806 04:33:53.809517 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:53.809520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:53.809564 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:53.809626 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:53.809885 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:53.810369 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:53.811139 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:53.811364 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:53.811446 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:53.812246 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:53.812250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:53.812253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:53.812267 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:53.812294 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:53.812306 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:53.812383 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:53.812542 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 04:33:53.812574 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:33:53.813406 26519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } }
I20260806 04:33:53.813773 26519 leader_election.cc:302] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19aeb0cea9f24d47a5a26fe3f8c6d7e3; no voters:
I20260806 04:33:53.814085 26519 leader_election.cc:288] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:53.814180 26531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:53.814395 26531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 19aeb0cea9f24d47a5a26fe3f8c6d7e3, State: Running, Role: LEADER
I20260806 04:33:53.814719 26531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } }
I20260806 04:33:53.815227 26519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:53.816270 26535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } } }
I20260806 04:33:53.816413 26535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:53.816697 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19aeb0cea9f24d47a5a26fe3f8c6d7e3. Latest consensus state: current_term: 1 leader_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19aeb0cea9f24d47a5a26fe3f8c6d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43225 } } }
I20260806 04:33:53.816803 26545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:53.816864 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:53.819953 26545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:53.825667 26545 catalog_manager.cc:1347] Generated new cluster ID: 5bdeefd215f54a1ea54158ff287bd8cf
I20260806 04:33:53.825781 26545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:53.832190 26545 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:53.833436 26545 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:53.843416 26545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19aeb0cea9f24d47a5a26fe3f8c6d7e3: Generated new TSK 0
I20260806 04:33:53.844276 26545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:53.856431 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26417
W20260806 04:33:53.867338 26400 connection.cc:535] client connection to 127.0.0.1:43225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43225): Connection reset by peer (error 104)
2026-08-06T04:33:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:53 dist-test-slave-ncp9 krb5kdc[26564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:53 dist-test-slave-ncp9 krb5kdc[26564](info): set up 2 sockets
Aug 06 04:33:53 dist-test-slave-ncp9 krb5kdc[26564](info): commencing operation
krb5kdc: starting...
W20260806 04:33:54.956323 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:55 dist-test-slave-ncp9 krb5kdc[26564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:33:55.084214 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:55.195765 26579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:55.196030 26579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:55.196069 26579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:55.200001 26579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:55.200060 26579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:55.200078 26579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:55.200093 26579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:55.200106 26579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:55.205235 26579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:55.206238 26579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:55.207198 26579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:55.215611 26589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:55.215636 26584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:55.215623 26585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:55.215695 26579 server_base.cc:1034] running on GCE node
I20260806 04:33:55.216379 26579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:55.217523 26579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:55.218685 26579 hybrid_clock.cc:648] HybridClock initialized: now 1785990835218662 us; error 37 us; skew 500 ppm
Aug 06 04:33:55 dist-test-slave-ncp9 krb5kdc[26564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:55.221405 26579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:33:55.222328 26579 webserver.cc:460] Webserver started at http://127.0.0.1:40567/ using document root <none> and password file <none>
I20260806 04:33:55.222873 26579 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:55.222939 26579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:55.223157 26579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:55.225256 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "989fc2f8824f46f7bc7b7e1487054030"
format_stamp: "Formatted at 2026-08-06 04:33:55 on dist-test-slave-ncp9"
server_key: "4291947326652049e10bc559eb9343fb"
server_key_iv: "0e74e58c3bc4ef586be0ac318c12d530"
server_key_version: "encryptionkey@0"
I20260806 04:33:55.225781 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "989fc2f8824f46f7bc7b7e1487054030"
format_stamp: "Formatted at 2026-08-06 04:33:55 on dist-test-slave-ncp9"
server_key: "4291947326652049e10bc559eb9343fb"
server_key_iv: "0e74e58c3bc4ef586be0ac318c12d530"
server_key_version: "encryptionkey@0"
I20260806 04:33:55.229440 26579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:55.231845 26602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:55.233129 26579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:33:55.233254 26579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "989fc2f8824f46f7bc7b7e1487054030"
format_stamp: "Formatted at 2026-08-06 04:33:55 on dist-test-slave-ncp9"
server_key: "4291947326652049e10bc559eb9343fb"
server_key_iv: "0e74e58c3bc4ef586be0ac318c12d530"
server_key_version: "encryptionkey@0"
I20260806 04:33:55.233345 26579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:55.237771 26579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:55.240693 26579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:55.240854 26579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:55.247407 26579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38467
I20260806 04:33:55.247419 26679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38467 every 8 connection(s)
I20260806 04:33:55.248293 26579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:55.250609 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26579
I20260806 04:33:55.250736 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:55.251044 29048 external_mini_cluster.cc:1428] Setting key 68bbbe590c4f0a63cb21ef73c1b969d1
I20260806 04:33:55.251296 26680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:55.255909 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:55.252336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:41447 (local address 127.0.0.1:38467)
0806 04:33:55.252727 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:55.252743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:55.252778 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:33:55.253436 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:55.253452 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:55.253726 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:55.253991 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:55.254761 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:55.254766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:55.254778 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:55.254788 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:55.254839 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:55.254857 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:55.254955 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:55.255059 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:33:55.255207 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260806 04:33:55.257752 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030: Bootstrap starting.
I20260806 04:33:55.258510 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:55.260398 26680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:55.261238 26680 log.cc:824] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:55.263022 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:55.260795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.213:36505 (local address 127.0.0.1:38467)
0806 04:33:55.260939 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:55.260944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:55.261794 (+ 850us) server_negotiation.cc:407] Connection header received
0806 04:33:55.261896 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:55.261899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:55.261932 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:55.261994 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:55.262666 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:55.262669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:55.262672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:55.262684 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:55.262708 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:55.262718 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:55.262787 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:55.262870 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 04:33:55.262910 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:33:55.263250 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030: No bootstrap required, opened a new log
I20260806 04:33:55.266945 26680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } }
I20260806 04:33:55.267175 26680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:55.267212 26680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989fc2f8824f46f7bc7b7e1487054030, State: Initialized, Role: FOLLOWER
I20260806 04:33:55.267619 26680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } }
I20260806 04:33:55.267755 26680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:55.267818 26680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:55.267917 26680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:55.268817 26680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } }
I20260806 04:33:55.269202 26680 leader_election.cc:302] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 989fc2f8824f46f7bc7b7e1487054030; no voters:
I20260806 04:33:55.269553 26680 leader_election.cc:288] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:55.269644 26692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:55.269873 26692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [term 1 LEADER]: Becoming Leader. State: Replica: 989fc2f8824f46f7bc7b7e1487054030, State: Running, Role: LEADER
I20260806 04:33:55.270236 26692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } }
I20260806 04:33:55.270833 26680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:55.271848 26694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989fc2f8824f46f7bc7b7e1487054030. Latest consensus state: current_term: 1 leader_uuid: "989fc2f8824f46f7bc7b7e1487054030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } } }
I20260806 04:33:55.271991 26694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:55.272257 26693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "989fc2f8824f46f7bc7b7e1487054030" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989fc2f8824f46f7bc7b7e1487054030" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38467 } } }
I20260806 04:33:55.272359 26693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:55.272372 26705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:55.275465 26705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:55.280882 26705 catalog_manager.cc:1347] Generated new cluster ID: 4e562db863d6479a9c059dea7e19d797
I20260806 04:33:55.280975 26705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:55.289996 26705 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:55.290293 26705 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:55.295688 26705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 989fc2f8824f46f7bc7b7e1487054030: Generated new TSK 0
I20260806 04:33:55.296447 26705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:55.299247 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26579
2026-08-06T04:33:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:55 dist-test-slave-ncp9 krb5kdc[26720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:55 dist-test-slave-ncp9 krb5kdc[26720](info): set up 2 sockets
Aug 06 04:33:55 dist-test-slave-ncp9 krb5kdc[26720](info): commencing operation
krb5kdc: starting...
W20260806 04:33:56.408818 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:56.530987 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33361
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:56.641497 26738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:56.641763 26738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:56.641819 26738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:56.645689 26738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:56.645744 26738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:56.645771 26738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:56.645790 26738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:56.645812 26738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:56.651229 26738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:56.652240 26738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:56.653239 26738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:33:56.661538 26738 server_base.cc:1034] running on GCE node
W20260806 04:33:56.661495 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:56.661469 26748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:56.661474 26751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:56.662216 26738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:56.663295 26738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:56.664453 26738 hybrid_clock.cc:648] HybridClock initialized: now 1785990836664433 us; error 31 us; skew 500 ppm
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990836, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:56.667024 26738 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:56.667868 26738 webserver.cc:460] Webserver started at http://127.28.94.62:37013/ using document root <none> and password file <none>
I20260806 04:33:56.668414 26738 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:56.668475 26738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:56.668648 26738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:56.670598 26738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "e48d410f8bca420aa5d013d595440d36"
format_stamp: "Formatted at 2026-08-06 04:33:56 on dist-test-slave-ncp9"
server_key: "399528f9de047ac4228ac46805da0ccc"
server_key_iv: "5c8c1da92a827e4b5e4998826d03bf20"
server_key_version: "encryptionkey@0"
I20260806 04:33:56.671078 26738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "e48d410f8bca420aa5d013d595440d36"
format_stamp: "Formatted at 2026-08-06 04:33:56 on dist-test-slave-ncp9"
server_key: "399528f9de047ac4228ac46805da0ccc"
server_key_iv: "5c8c1da92a827e4b5e4998826d03bf20"
server_key_version: "encryptionkey@0"
I20260806 04:33:56.674602 26738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 04:33:56.676909 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:56.678112 26738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:33:56.678212 26738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "e48d410f8bca420aa5d013d595440d36"
format_stamp: "Formatted at 2026-08-06 04:33:56 on dist-test-slave-ncp9"
server_key: "399528f9de047ac4228ac46805da0ccc"
server_key_iv: "5c8c1da92a827e4b5e4998826d03bf20"
server_key_version: "encryptionkey@0"
I20260806 04:33:56.678294 26738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:56.699319 26738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:56.702370 26738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:56.702544 26738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:56.709779 26738 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:33361
I20260806 04:33:56.709805 26827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:33361 every 8 connection(s)
I20260806 04:33:56.710741 26738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:56.713690 26828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:56.717195 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26738
I20260806 04:33:56.717317 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:56.717581 29048 external_mini_cluster.cc:1428] Setting key 13bf02d3f42e50ee08a0ee422ff026e6
I20260806 04:33:56.719666 26828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: Bootstrap starting.
I20260806 04:33:56.722237 26828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:56.723018 26828 log.cc:824] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:56.725234 26828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: No bootstrap required, opened a new log
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:33:56.728973 26828 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } }
I20260806 04:33:56.729208 26828 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:56.729265 26828 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48d410f8bca420aa5d013d595440d36, State: Initialized, Role: FOLLOWER
I20260806 04:33:56.729723 26828 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } }
I20260806 04:33:56.729862 26828 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:56.729929 26828 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:56.730026 26828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:56.730990 26828 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } }
I20260806 04:33:56.731326 26828 leader_election.cc:302] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48d410f8bca420aa5d013d595440d36; no voters:
I20260806 04:33:56.731626 26828 leader_election.cc:288] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:56.731739 26836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:56.731886 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:56.718991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43307 (local address 127.28.94.62:33361)
0806 04:33:56.719418 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:56.719434 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:56.719471 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:33:56.720124 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:56.720145 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:56.720487 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:56.720769 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:56.721721 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:56.722806 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:56.723835 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:56.724296 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:56.724423 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:56.727114 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:56.727119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:56.727133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:56.727162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:56.728726 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:56.729262 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:56.729269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:56.729275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:56.729414 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:56.729739 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:56.729743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:56.729745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:56.730219 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:56.730415 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:56.730771 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 04:33:56.730985 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":117,"threads_started":1}
I20260806 04:33:56.731972 26836 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [term 1 LEADER]: Becoming Leader. State: Replica: e48d410f8bca420aa5d013d595440d36, State: Running, Role: LEADER
I20260806 04:33:56.732488 26836 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } }
I20260806 04:33:56.733000 26828 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:56.734197 26838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48d410f8bca420aa5d013d595440d36. Latest consensus state: current_term: 1 leader_uuid: "e48d410f8bca420aa5d013d595440d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } } }
I20260806 04:33:56.734550 26838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:56.734443 26837 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48d410f8bca420aa5d013d595440d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48d410f8bca420aa5d013d595440d36" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33361 } } }
I20260806 04:33:56.734822 26837 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:56.735613 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 04:33:56.737927 26861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:33:56.738054 26861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:33:56.738178 26857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:56.739055 26857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:56.745028 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:56.737692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32827 (local address 127.28.94.62:33361)
0806 04:33:56.737798 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:56.737803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:56.738015 (+ 212us) server_negotiation.cc:407] Connection header received
0806 04:33:56.738121 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:56.738124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:56.738169 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:56.738232 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:56.738856 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:56.739368 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:56.740132 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:56.740370 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:56.740455 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:56.741783 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:56.741787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:56.741789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:33:56.741814 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:56.743069 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:56.743633 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:56.743637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:56.743639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:56.743729 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:33:56.744107 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:33:56.744111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:33:56.744113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:33:56.744324 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:56.744464 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:56.744763 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 04:33:56.744859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:33:56.745312 26857 catalog_manager.cc:1347] Generated new cluster ID: d80b21f5a1d44ae4aa525b764ea06775
I20260806 04:33:56.745394 26857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:56.774967 26857 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:56.775945 26857 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:56.783921 26857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48d410f8bca420aa5d013d595440d36: Generated new TSK 0
I20260806 04:33:56.784585 26857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:56.795238 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26738
W20260806 04:33:56.811015 26716 connection.cc:535] client connection to 127.28.94.62:33361 recv error: Network error: failed to read from TLS socket (remote: 127.28.94.62:33361): Connection reset by peer (error 104)
2026-08-06T04:33:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26877](info): set up 2 sockets
Aug 06 04:33:56 dist-test-slave-ncp9 krb5kdc[26877](info): commencing operation
krb5kdc: starting...
W20260806 04:33:57.892927 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:57 dist-test-slave-ncp9 krb5kdc[26877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:58.014040 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33709
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:58.128393 26893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:58.128674 26893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:58.128727 26893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:58.132630 26893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:58.132692 26893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:58.132717 26893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:58.132738 26893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:58.132758 26893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:58.137909 26893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:58.138968 26893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:58.139998 26893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:58.148777 26903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:58.148790 26906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:58.148834 26893 server_base.cc:1034] running on GCE node
W20260806 04:33:58.148790 26904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:58.149616 26893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:58.150697 26893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:58.151840 26893 hybrid_clock.cc:648] HybridClock initialized: now 1785990838151824 us; error 34 us; skew 500 ppm
Aug 06 04:33:58 dist-test-slave-ncp9 krb5kdc[26877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:58.154596 26893 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:58.155462 26893 webserver.cc:460] Webserver started at http://127.28.94.62:37763/ using document root <none> and password file <none>
I20260806 04:33:58.155997 26893 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:58.156055 26893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:58.156255 26893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:58.158304 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "089d4d71fbb542baafa2e00b1448cff2"
format_stamp: "Formatted at 2026-08-06 04:33:58 on dist-test-slave-ncp9"
server_key: "451c5393d12b54621f3d4a3eef1c6b37"
server_key_iv: "f31c398ad915a42b6479bbde11ba7a7d"
server_key_version: "encryptionkey@0"
I20260806 04:33:58.158847 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "089d4d71fbb542baafa2e00b1448cff2"
format_stamp: "Formatted at 2026-08-06 04:33:58 on dist-test-slave-ncp9"
server_key: "451c5393d12b54621f3d4a3eef1c6b37"
server_key_iv: "f31c398ad915a42b6479bbde11ba7a7d"
server_key_version: "encryptionkey@0"
I20260806 04:33:58.162487 26893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:33:58.164804 26917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:58.166064 26893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:33:58.166172 26893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "089d4d71fbb542baafa2e00b1448cff2"
format_stamp: "Formatted at 2026-08-06 04:33:58 on dist-test-slave-ncp9"
server_key: "451c5393d12b54621f3d4a3eef1c6b37"
server_key_iv: "f31c398ad915a42b6479bbde11ba7a7d"
server_key_version: "encryptionkey@0"
I20260806 04:33:58.166270 26893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:58.187592 26893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:58.190698 26893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:58.190881 26893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:58.197690 26893 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:33709
I20260806 04:33:58.197726 26992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:33709 every 8 connection(s)
I20260806 04:33:58.198633 26893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:58.200165 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 26893
I20260806 04:33:58.200285 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:58.200544 29048 external_mini_cluster.cc:1428] Setting key 6f3679b9fb017e4835176014c536411d
I20260806 04:33:58.201771 26993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:58.207947 26993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2: Bootstrap starting.
I20260806 04:33:58.209273 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:58.201935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54249 (local address 127.28.94.62:33709)
0806 04:33:58.202354 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:58.202370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:58.202406 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:33:58.203027 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:58.203043 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:58.203364 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:58.203626 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:58.203981 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:58.205001 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:58.206158 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:58.206600 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:58.206715 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:58.207606 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:58.207611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:58.207624 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:58.207997 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:58.208048 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:58.208076 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:58.208194 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:58.208362 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 04:33:58.208495 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
I20260806 04:33:58.210489 26993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:58.211297 26993 log.cc:824] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:58.212200 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:58.213379 26993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2: No bootstrap required, opened a new log
I20260806 04:33:58.216476 26993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } }
I20260806 04:33:58.216707 26993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:58.216766 26993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089d4d71fbb542baafa2e00b1448cff2, State: Initialized, Role: FOLLOWER
I20260806 04:33:58.217231 26993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } }
I20260806 04:33:58.217365 26993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:58.217432 26993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:58.217528 26993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:58.218292 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:58.214260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43063 (local address 127.28.94.62:33709)
0806 04:33:58.214365 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:58.214369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:58.214514 (+ 145us) server_negotiation.cc:407] Connection header received
0806 04:33:58.214623 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:58.214626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:58.214673 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:58.214733 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:58.214994 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:58.215843 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:58.216634 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:33:58.216977 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:33:58.217062 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:33:58.217866 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:58.217869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:58.217872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:58.217884 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:58.217909 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:58.217919 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:58.217989 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:58.218126 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 04:33:58.218163 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:33:58.218503 26993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } }
I20260806 04:33:58.218840 26993 leader_election.cc:302] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089d4d71fbb542baafa2e00b1448cff2; no voters:
I20260806 04:33:58.219166 26993 leader_election.cc:288] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:58.219272 27007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:58.219521 27007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [term 1 LEADER]: Becoming Leader. State: Replica: 089d4d71fbb542baafa2e00b1448cff2, State: Running, Role: LEADER
I20260806 04:33:58.219858 27007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } }
I20260806 04:33:58.220454 26993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:58.221395 27009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089d4d71fbb542baafa2e00b1448cff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } } }
I20260806 04:33:58.221521 27009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:58.221874 27008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089d4d71fbb542baafa2e00b1448cff2. Latest consensus state: current_term: 1 leader_uuid: "089d4d71fbb542baafa2e00b1448cff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089d4d71fbb542baafa2e00b1448cff2" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33709 } } }
I20260806 04:33:58.221998 27008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:58.221912 27019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:58.225085 27019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:58.230373 27019 catalog_manager.cc:1347] Generated new cluster ID: 3fdd32cb6bb64b14abe763dc6418113d
I20260806 04:33:58.230446 27019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:58.248919 27019 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:58.250053 27019 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:58.265375 27019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089d4d71fbb542baafa2e00b1448cff2: Generated new TSK 0
I20260806 04:33:58.266095 27019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:58.271914 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 26893
W20260806 04:33:58.281097 26875 connection.cc:535] client connection to 127.28.94.62:33709 recv error: Network error: failed to read from TLS socket (remote: 127.28.94.62:33709): Connection reset by peer (error 104)
2026-08-06T04:33:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:58 dist-test-slave-ncp9 krb5kdc[27039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:58 dist-test-slave-ncp9 krb5kdc[27039](info): set up 2 sockets
Aug 06 04:33:58 dist-test-slave-ncp9 krb5kdc[27039](info): commencing operation
krb5kdc: starting...
W20260806 04:33:59.373209 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:33:59 dist-test-slave-ncp9 krb5kdc[27039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:33:59.500918 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39393
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:44473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:39393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:33:59.614707 27056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:33:59.614967 27056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:33:59.615006 27056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:33:59.618960 27056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:33:59.619021 27056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:33:59.619040 27056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:33:59.619053 27056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:33:59.619067 27056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:33:59.624316 27056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:44473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:39393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:39393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:33:59.625449 27056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:33:59.626451 27056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:33:59.634843 27063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:59.634831 27062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:33:59.634835 27066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:33:59.635047 27056 server_base.cc:1034] running on GCE node
I20260806 04:33:59.635706 27056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:33:59.636808 27056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:33:59.637974 27056 hybrid_clock.cc:648] HybridClock initialized: now 1785990839637947 us; error 30 us; skew 500 ppm
Aug 06 04:33:59 dist-test-slave-ncp9 krb5kdc[27039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:33:59.644542 27056 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:33:59.645510 27056 webserver.cc:460] Webserver started at http://127.28.94.62:38235/ using document root <none> and password file <none>
I20260806 04:33:59.646075 27056 fs_manager.cc:359] Metadata directory not provided
I20260806 04:33:59.646144 27056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:33:59.646349 27056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:33:59.648366 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "e5df152519fe418faa7049c0e23ce931"
format_stamp: "Formatted at 2026-08-06 04:33:59 on dist-test-slave-ncp9"
server_key: "bbeab6faf47f08f79789a247d2376fcb"
server_key_iv: "f37d4f0c0a5f30a690c70479d2065521"
server_key_version: "encryptionkey@0"
I20260806 04:33:59.648919 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "e5df152519fe418faa7049c0e23ce931"
format_stamp: "Formatted at 2026-08-06 04:33:59 on dist-test-slave-ncp9"
server_key: "bbeab6faf47f08f79789a247d2376fcb"
server_key_iv: "f37d4f0c0a5f30a690c70479d2065521"
server_key_version: "encryptionkey@0"
I20260806 04:33:59.652489 27056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:33:59.654855 27073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:33:59.656105 27056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:33:59.656231 27056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "e5df152519fe418faa7049c0e23ce931"
format_stamp: "Formatted at 2026-08-06 04:33:59 on dist-test-slave-ncp9"
server_key: "bbeab6faf47f08f79789a247d2376fcb"
server_key_iv: "f37d4f0c0a5f30a690c70479d2065521"
server_key_version: "encryptionkey@0"
I20260806 04:33:59.656329 27056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:33:59.660878 27056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:33:59.663954 27056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:33:59.664132 27056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:33:59.671191 27056 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:39393
I20260806 04:33:59.671208 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:39393 every 8 connection(s)
I20260806 04:33:59.672113 27056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:33:59.674985 27153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:33:59.678045 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27056
I20260806 04:33:59.678184 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:33:59.678434 29048 external_mini_cluster.cc:1428] Setting key 91c09cd0de5522ddbda3886df81d45e1
I20260806 04:33:59.681079 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: Bootstrap starting.
I20260806 04:33:59.683776 27153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: Neither blocks nor log segments found. Creating new log.
I20260806 04:33:59.684008 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:59.679964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51625 (local address 127.28.94.62:39393)
0806 04:33:59.680397 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:59.680417 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:59.680460 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:33:59.681304 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:59.681329 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:59.681693 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:59.682041 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:59.682821 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:59.682826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:59.682838 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:59.682849 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:59.682915 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:59.682936 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:59.683065 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:59.683138 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 04:33:59.683294 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260806 04:33:59.684553 27153 log.cc:824] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: Log is configured to *not* fsync() on all Append() calls
I20260806 04:33:59.686600 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: No bootstrap required, opened a new log
I20260806 04:33:59.686707 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:33:59.690114 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:33:59.688615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37361 (local address 127.28.94.62:39393)
0806 04:33:59.688736 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:33:59.688740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:33:59.688920 (+ 180us) server_negotiation.cc:407] Connection header received
0806 04:33:59.689035 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:33:59.689038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:33:59.689074 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:33:59.689156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:33:59.689756 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:33:59.689759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:33:59.689762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:33:59.689772 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:33:59.689798 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:33:59.689811 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:33:59.689893 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:33:59.689940 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 04:33:59.689976 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 04:33:59.690678 27153 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } }
I20260806 04:33:59.690912 27153 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:33:59.690997 27153 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5df152519fe418faa7049c0e23ce931, State: Initialized, Role: FOLLOWER
I20260806 04:33:59.691422 27153 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } }
I20260806 04:33:59.691555 27153 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:33:59.691623 27153 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:33:59.691723 27153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:33:59.692655 27153 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } }
I20260806 04:33:59.693033 27153 leader_election.cc:302] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5df152519fe418faa7049c0e23ce931; no voters:
I20260806 04:33:59.693333 27153 leader_election.cc:288] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:33:59.693482 27166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:33:59.693776 27166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [term 1 LEADER]: Becoming Leader. State: Replica: e5df152519fe418faa7049c0e23ce931, State: Running, Role: LEADER
I20260806 04:33:59.694108 27166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } }
I20260806 04:33:59.694644 27153 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:33:59.695787 27168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5df152519fe418faa7049c0e23ce931. Latest consensus state: current_term: 1 leader_uuid: "e5df152519fe418faa7049c0e23ce931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } } }
I20260806 04:33:59.695914 27168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [sys.catalog]: This master's current role is: LEADER
I20260806 04:33:59.696087 27167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5df152519fe418faa7049c0e23ce931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5df152519fe418faa7049c0e23ce931" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 39393 } } }
I20260806 04:33:59.696244 27167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931 [sys.catalog]: This master's current role is: LEADER
W20260806 04:33:59.699594 27183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:33:59.699666 27183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:33:59.699756 27185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:33:59.700449 27185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:33:59.705978 27185 catalog_manager.cc:1347] Generated new cluster ID: d925d2bfad7f4011909e9bd340526bdb
I20260806 04:33:59.706067 27185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:33:59.715247 27185 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:33:59.715565 27185 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:33:59.721912 27185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5df152519fe418faa7049c0e23ce931: Generated new TSK 0
I20260806 04:33:59.722599 27185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:33:59.730628 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27056
2026-08-06T04:33:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:33:59 dist-test-slave-ncp9 krb5kdc[27192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:33:59 dist-test-slave-ncp9 krb5kdc[27192](info): set up 2 sockets
Aug 06 04:33:59 dist-test-slave-ncp9 krb5kdc[27192](info): commencing operation
krb5kdc: starting...
W20260806 04:34:00.840627 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:34:00 dist-test-slave-ncp9 krb5kdc[27192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:34:00.964264 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43427
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:34:01.077013 27208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:01.077252 27208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:01.077292 27208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:01.081406 27208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:01.081466 27208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:01.081485 27208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:01.081499 27208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:01.081513 27208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:01.086707 27208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:43427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:01.087687 27208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:01.088640 27208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:01.097402 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:01.097425 27215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:01.097421 27223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:01.097697 27208 server_base.cc:1034] running on GCE node
I20260806 04:34:01.098239 27208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:01.099287 27208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:01.100437 27208 hybrid_clock.cc:648] HybridClock initialized: now 1785990841100414 us; error 32 us; skew 500 ppm
Aug 06 04:34:01 dist-test-slave-ncp9 krb5kdc[27192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:01.103147 27208 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:34:01.104015 27208 webserver.cc:460] Webserver started at http://127.28.94.62:34673/ using document root <none> and password file <none>
I20260806 04:34:01.104543 27208 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:01.104641 27208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:01.104849 27208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:01.106920 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "1639636a2605409badea92bddda63d40"
format_stamp: "Formatted at 2026-08-06 04:34:01 on dist-test-slave-ncp9"
server_key: "c1d0eff98e1c84ec234cbc97b2829cbb"
server_key_iv: "812c296bd435f5b8b9e2671139f96969"
server_key_version: "encryptionkey@0"
I20260806 04:34:01.107465 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "1639636a2605409badea92bddda63d40"
format_stamp: "Formatted at 2026-08-06 04:34:01 on dist-test-slave-ncp9"
server_key: "c1d0eff98e1c84ec234cbc97b2829cbb"
server_key_iv: "812c296bd435f5b8b9e2671139f96969"
server_key_version: "encryptionkey@0"
I20260806 04:34:01.111095 27208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:34:01.113385 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:01.114542 27208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:34:01.114665 27208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "1639636a2605409badea92bddda63d40"
format_stamp: "Formatted at 2026-08-06 04:34:01 on dist-test-slave-ncp9"
server_key: "c1d0eff98e1c84ec234cbc97b2829cbb"
server_key_iv: "812c296bd435f5b8b9e2671139f96969"
server_key_version: "encryptionkey@0"
I20260806 04:34:01.114754 27208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:01.134552 27208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:01.137584 27208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:01.137766 27208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:01.144886 27208 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:43427
I20260806 04:34:01.144945 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:43427 every 8 connection(s)
I20260806 04:34:01.146050 27208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:34:01.148836 27311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:01.151103 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27208
I20260806 04:34:01.151225 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:34:01.151473 29048 external_mini_cluster.cc:1428] Setting key ebfac5d3a436aec6096696bd98a8b691
I20260806 04:34:01.154769 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: Bootstrap starting.
I20260806 04:34:01.157325 27311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:01.158245 27311 log.cc:824] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:34:01 dist-test-slave-ncp9 krb5kdc[27192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:34:01.160468 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: No bootstrap required, opened a new log
I20260806 04:34:01.164873 27311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } }
I20260806 04:34:01.165159 27311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:34:01.165220 27311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1639636a2605409badea92bddda63d40, State: Initialized, Role: FOLLOWER
I20260806 04:34:01.165725 27311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } }
I20260806 04:34:01.165886 27311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:34:01.165908 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:01.152705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37627 (local address 127.28.94.62:43427)
0806 04:34:01.153112 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:01.153127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:01.153163 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:34:01.153819 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:01.153837 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:01.154174 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:01.154442 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:01.155346 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:01.156311 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:01.157497 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:01.158190 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:01.158352 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:01.160980 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:01.160986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:01.161003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:01.161033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:01.162913 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:01.163313 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:01.163320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:01.163325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:01.163466 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:01.163702 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:01.163705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:01.163707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:01.164227 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:01.164428 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:01.164601 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:34:01.164919 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260806 04:34:01.165958 27311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:34:01.166045 27311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:01.167006 27311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } }
I20260806 04:34:01.167362 27311 leader_election.cc:302] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1639636a2605409badea92bddda63d40; no voters:
I20260806 04:34:01.167719 27311 leader_election.cc:288] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:34:01.167786 27319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:34:01.168028 27319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [term 1 LEADER]: Becoming Leader. State: Replica: 1639636a2605409badea92bddda63d40, State: Running, Role: LEADER
I20260806 04:34:01.168394 27319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } }
I20260806 04:34:01.168804 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:34:01.169106 27311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:01.171197 27320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1639636a2605409badea92bddda63d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } } }
I20260806 04:34:01.171644 27320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:01.171402 27321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1639636a2605409badea92bddda63d40. Latest consensus state: current_term: 1 leader_uuid: "1639636a2605409badea92bddda63d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1639636a2605409badea92bddda63d40" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43427 } } }
I20260806 04:34:01.172020 27321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40 [sys.catalog]: This master's current role is: LEADER
W20260806 04:34:01.174250 27343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:01.174335 27343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:01.174412 27346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:34:01.175241 27346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:34:01.180644 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:01.171693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45055 (local address 127.28.94.62:43427)
0806 04:34:01.171952 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:01.171956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:01.172174 (+ 218us) server_negotiation.cc:407] Connection header received
0806 04:34:01.172303 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:01.172307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:01.172352 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:01.172426 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:01.173277 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:01.174031 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:01.175120 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:01.175465 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:01.175571 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:01.177196 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:01.177200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:01.177203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:01.177230 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:01.178821 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:01.179334 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:01.179338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:01.179340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:01.179426 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:01.179736 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:01.179739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:01.179741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:01.179974 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:01.180114 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:01.180363 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:34:01.180483 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260806 04:34:01.181202 27346 catalog_manager.cc:1347] Generated new cluster ID: e5c4dea16c9f449c990cf610e1e5786b
I20260806 04:34:01.181267 27346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:34:01.201654 27346 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:34:01.202591 27346 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:34:01.206825 27346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1639636a2605409badea92bddda63d40: Generated new TSK 0
I20260806 04:34:01.207430 27346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:34:01.215829 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27208
2026-08-06T04:34:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:34:01 dist-test-slave-ncp9 krb5kdc[27356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:34:01 dist-test-slave-ncp9 krb5kdc[27356](info): set up 2 sockets
Aug 06 04:34:01 dist-test-slave-ncp9 krb5kdc[27356](info): commencing operation
krb5kdc: starting...
W20260806 04:34:02.328267 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:34:02 dist-test-slave-ncp9 krb5kdc[27356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:34:02.451668 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40011
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:45323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:34:02.563709 27377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:02.563967 27377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:02.564034 27377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:02.567924 27377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:02.567983 27377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:02.568009 27377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:02.568039 27377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:02.568068 27377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:02.573181 27377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:45323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:40011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:02.574178 27377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:02.575172 27377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:02.583442 27384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:02.583514 27390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:02.583509 27377 server_base.cc:1034] running on GCE node
W20260806 04:34:02.583649 27386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:02.584194 27377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:02.585283 27377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:02.586442 27377 hybrid_clock.cc:648] HybridClock initialized: now 1785990842586403 us; error 48 us; skew 500 ppm
Aug 06 04:34:02 dist-test-slave-ncp9 krb5kdc[27356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990842, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:02.589264 27377 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:34:02.590144 27377 webserver.cc:460] Webserver started at http://127.28.94.62:35649/ using document root <none> and password file <none>
I20260806 04:34:02.590677 27377 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:02.590737 27377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:02.590977 27377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:02.593042 27377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "20987e811f2e41efbc687c914fb4c59f"
format_stamp: "Formatted at 2026-08-06 04:34:02 on dist-test-slave-ncp9"
server_key: "713f3e400a4e0142518cd12cae9a0167"
server_key_iv: "81bc141dcd8566bca359f3312e089672"
server_key_version: "encryptionkey@0"
I20260806 04:34:02.593592 27377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "20987e811f2e41efbc687c914fb4c59f"
format_stamp: "Formatted at 2026-08-06 04:34:02 on dist-test-slave-ncp9"
server_key: "713f3e400a4e0142518cd12cae9a0167"
server_key_iv: "81bc141dcd8566bca359f3312e089672"
server_key_version: "encryptionkey@0"
I20260806 04:34:02.597261 27377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:34:02.599694 27397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:02.601028 27377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:34:02.601166 27377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "20987e811f2e41efbc687c914fb4c59f"
format_stamp: "Formatted at 2026-08-06 04:34:02 on dist-test-slave-ncp9"
server_key: "713f3e400a4e0142518cd12cae9a0167"
server_key_iv: "81bc141dcd8566bca359f3312e089672"
server_key_version: "encryptionkey@0"
I20260806 04:34:02.601262 27377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:02.612578 27377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:02.615410 27377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:02.615582 27377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:02.622386 27377 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:40011
I20260806 04:34:02.622430 27470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:40011 every 8 connection(s)
I20260806 04:34:02.623329 27377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:34:02.626217 27471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:02.628517 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27377
I20260806 04:34:02.628648 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:34:02.628945 29048 external_mini_cluster.cc:1428] Setting key 5b15146a20642b687ba6fb0684b02b4d
I20260806 04:34:02.632761 27471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f: Bootstrap starting.
I20260806 04:34:02.635444 27471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:02.636361 27471 log.cc:824] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:02.638669 27471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f: No bootstrap required, opened a new log
I20260806 04:34:02.638917 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:02.630369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55783 (local address 127.28.94.62:40011)
0806 04:34:02.630803 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:02.630823 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:02.630868 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:34:02.631658 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:02.631684 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:02.632125 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:02.632481 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:02.632942 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:02.634392 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:02.635328 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:02.636006 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:02.636178 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:02.636958 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:02.636963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:02.636976 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:34:02.637455 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:02.637520 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:34:02.637551 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:02.637650 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:02.637776 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 04:34:02.637978 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
I20260806 04:34:02.641590 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:34:02.642629 27471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } }
I20260806 04:34:02.642908 27471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:34:02.642972 27471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20987e811f2e41efbc687c914fb4c59f, State: Initialized, Role: FOLLOWER
I20260806 04:34:02.643443 27471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } }
I20260806 04:34:02.643621 27471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:34:02.643692 27471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:34:02.643788 27471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:02.644876 27471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } }
I20260806 04:34:02.645226 27471 leader_election.cc:302] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20987e811f2e41efbc687c914fb4c59f; no voters:
I20260806 04:34:02.645569 27471 leader_election.cc:288] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:34:02.645663 27486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:34:02.645884 27486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [term 1 LEADER]: Becoming Leader. State: Replica: 20987e811f2e41efbc687c914fb4c59f, State: Running, Role: LEADER
I20260806 04:34:02.646204 27486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } }
I20260806 04:34:02.646819 27471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:02.648027 27489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20987e811f2e41efbc687c914fb4c59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } } }
I20260806 04:34:02.648169 27489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:02.648217 27474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:02.643771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39223 (local address 127.28.94.62:40011)
0806 04:34:02.643898 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:02.643903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:02.644138 (+ 235us) server_negotiation.cc:407] Connection header received
0806 04:34:02.644242 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:02.644244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:02.644280 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:02.644338 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:02.644604 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:02.645178 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:02.646244 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:02.646575 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:02.646679 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:02.647510 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:02.647514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:02.647517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:34:02.647530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:02.647555 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:34:02.647569 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:02.647639 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:02.648025 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 04:34:02.648092 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:34:02.648548 27499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:34:02.649871 27490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20987e811f2e41efbc687c914fb4c59f. Latest consensus state: current_term: 1 leader_uuid: "20987e811f2e41efbc687c914fb4c59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20987e811f2e41efbc687c914fb4c59f" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 40011 } } }
I20260806 04:34:02.649969 27490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:02.651760 27499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:34:02.657642 27499 catalog_manager.cc:1347] Generated new cluster ID: 569baabbd34f4194b6ce5818457941bb
I20260806 04:34:02.657698 27499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:34:02.666314 27499 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:34:02.667193 27499 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:34:02.671877 27499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20987e811f2e41efbc687c914fb4c59f: Generated new TSK 0
I20260806 04:34:02.672487 27499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:34:02.679750 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27377
2026-08-06T04:34:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:34:02 dist-test-slave-ncp9 krb5kdc[27514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:34:02 dist-test-slave-ncp9 krb5kdc[27514](info): set up 2 sockets
Aug 06 04:34:02 dist-test-slave-ncp9 krb5kdc[27514](info): commencing operation
krb5kdc: starting...
W20260806 04:34:03.781371 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:34:03 dist-test-slave-ncp9 krb5kdc[27514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:34:03.907307 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37539
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:36199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:34:04.022567 27535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:04.022841 27535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:04.022882 27535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:04.026819 27535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:04.026873 27535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:04.026891 27535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:04.026906 27535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:04.026919 27535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:04.032039 27535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:36199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:37539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:04.033221 27535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:04.034392 27535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:04.042851 27543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:04.042939 27548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:04.042863 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:04.043211 27535 server_base.cc:1034] running on GCE node
I20260806 04:34:04.043751 27535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:04.044844 27535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:04.046046 27535 hybrid_clock.cc:648] HybridClock initialized: now 1785990844046017 us; error 40 us; skew 500 ppm
Aug 06 04:34:04 dist-test-slave-ncp9 krb5kdc[27514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990844, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:04.048810 27535 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:34:04.049674 27535 webserver.cc:460] Webserver started at http://127.28.94.62:43317/ using document root <none> and password file <none>
I20260806 04:34:04.050206 27535 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:04.050272 27535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:04.050462 27535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:04.052418 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "4bb83c5bfa6642c78bf97f11b0613adc"
format_stamp: "Formatted at 2026-08-06 04:34:04 on dist-test-slave-ncp9"
server_key: "756a8865364886ed0b2ef6c9fd1fcca8"
server_key_iv: "4badbe887d5ca8a8f5586bb500355943"
server_key_version: "encryptionkey@0"
I20260806 04:34:04.053040 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "4bb83c5bfa6642c78bf97f11b0613adc"
format_stamp: "Formatted at 2026-08-06 04:34:04 on dist-test-slave-ncp9"
server_key: "756a8865364886ed0b2ef6c9fd1fcca8"
server_key_iv: "4badbe887d5ca8a8f5586bb500355943"
server_key_version: "encryptionkey@0"
I20260806 04:34:04.056522 27535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:34:04.058890 27556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:04.060030 27535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:04.060144 27535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "4bb83c5bfa6642c78bf97f11b0613adc"
format_stamp: "Formatted at 2026-08-06 04:34:04 on dist-test-slave-ncp9"
server_key: "756a8865364886ed0b2ef6c9fd1fcca8"
server_key_iv: "4badbe887d5ca8a8f5586bb500355943"
server_key_version: "encryptionkey@0"
I20260806 04:34:04.060236 27535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:04.064656 27535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:04.067665 27535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:04.067852 27535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:04.074945 27535 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:37539
I20260806 04:34:04.074977 27632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:37539 every 8 connection(s)
I20260806 04:34:04.075875 27535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:34:04.078794 27633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:04.083570 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27535
I20260806 04:34:04.083693 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:34:04.083933 29048 external_mini_cluster.cc:1428] Setting key 5f40a24f1c62acc72104dce3d735e682
I20260806 04:34:04.084703 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: Bootstrap starting.
I20260806 04:34:04.087232 27633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:04.088129 27633 log.cc:824] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:04.089157 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:04.085342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47975 (local address 127.28.94.62:37539)
0806 04:34:04.085755 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:04.085771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:04.085806 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:34:04.086428 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:04.086444 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:04.086718 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:04.086985 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:04.087808 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:04.087813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:04.087826 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:34:04.087837 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:04.087892 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:34:04.087912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:04.087993 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:04.088127 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 04:34:04.088278 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260806 04:34:04.090507 27633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: No bootstrap required, opened a new log
I20260806 04:34:04.092110 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:34:04.094408 27633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } }
I20260806 04:34:04.094705 27633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:34:04.094779 27633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb83c5bfa6642c78bf97f11b0613adc, State: Initialized, Role: FOLLOWER
I20260806 04:34:04.095223 27633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } }
I20260806 04:34:04.095396 27633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:34:04.095481 27633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:34:04.095594 27633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:04.095708 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:04.094125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41299 (local address 127.28.94.62:37539)
0806 04:34:04.094241 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:04.094245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:04.094437 (+ 192us) server_negotiation.cc:407] Connection header received
0806 04:34:04.094531 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:04.094535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:04.094573 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:04.094668 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:04.095280 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:04.095284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:04.095287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:34:04.095301 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:04.095333 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:34:04.095349 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:04.095447 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:04.095511 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 04:34:04.095554 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:34:04.096716 27633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } }
I20260806 04:34:04.097092 27633 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb83c5bfa6642c78bf97f11b0613adc; no voters:
I20260806 04:34:04.097365 27633 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:34:04.097457 27646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:34:04.097671 27646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [term 1 LEADER]: Becoming Leader. State: Replica: 4bb83c5bfa6642c78bf97f11b0613adc, State: Running, Role: LEADER
I20260806 04:34:04.098004 27646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } }
I20260806 04:34:04.098531 27633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:04.099937 27647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } } }
I20260806 04:34:04.100091 27647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:04.100526 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb83c5bfa6642c78bf97f11b0613adc. Latest consensus state: current_term: 1 leader_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb83c5bfa6642c78bf97f11b0613adc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 37539 } } }
I20260806 04:34:04.101450 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc [sys.catalog]: This master's current role is: LEADER
W20260806 04:34:04.104002 27668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:04.104092 27668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:04.104198 27656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:34:04.104954 27656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:34:04.110298 27656 catalog_manager.cc:1347] Generated new cluster ID: a441394a6f4b46a993364cf0a1f9fe2f
I20260806 04:34:04.110369 27656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:34:04.119091 27656 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:34:04.119360 27656 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:34:04.125801 27656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb83c5bfa6642c78bf97f11b0613adc: Generated new TSK 0
I20260806 04:34:04.126435 27656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:34:04.139070 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27535
2026-08-06T04:34:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1459 ms)
[----------] 18 tests from AuthTokenIssuingTest (26697 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:34:04 dist-test-slave-ncp9 krb5kdc[27675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:34:04 dist-test-slave-ncp9 krb5kdc[27675](info): set up 2 sockets
Aug 06 04:34:04 dist-test-slave-ncp9 krb5kdc[27675](info): commencing operation
krb5kdc: starting...
W20260806 04:34:05.260733 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:34:05.387487 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43927
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:34:05.500699 27697 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:05.501011 27697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:05.501048 27697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:05.501111 27697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:05.505095 27697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:05.505158 27697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:05.505184 27697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:05.505204 27697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:05.505223 27697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:05.510481 27697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:43927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:05.511579 27697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:05.512610 27697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:05.521139 27704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.521219 27697 server_base.cc:1034] running on GCE node
W20260806 04:34:05.521287 27709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:05.521147 27705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.521981 27697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:05.523064 27697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:05.524223 27697 hybrid_clock.cc:648] HybridClock initialized: now 1785990845524196 us; error 39 us; skew 500 ppm
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:05.531144 27697 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:34:05.532081 27697 webserver.cc:460] Webserver started at http://127.28.94.62:33289/ using document root <none> and password file <none>
I20260806 04:34:05.532624 27697 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:05.532691 27697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:05.532922 27697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:05.535014 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "04078c96a38f40b5bdedb43d07b1e0bc"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "20600d2ebab4d1ca0713af3dec988c85"
server_key_iv: "fb84848202446c4ac84177406868a905"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.535557 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "04078c96a38f40b5bdedb43d07b1e0bc"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "20600d2ebab4d1ca0713af3dec988c85"
server_key_iv: "fb84848202446c4ac84177406868a905"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.539316 27697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:34:05.541697 27717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:05.542898 27697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:05.543013 27697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "04078c96a38f40b5bdedb43d07b1e0bc"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "20600d2ebab4d1ca0713af3dec988c85"
server_key_iv: "fb84848202446c4ac84177406868a905"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.543118 27697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:05.559154 27697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:05.561920 27697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:05.562101 27697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:05.569291 27697 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:43927
I20260806 04:34:05.569550 27803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:43927 every 8 connection(s)
I20260806 04:34:05.570231 27697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:34:05.572980 27805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:05.574046 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27697
I20260806 04:34:05.574152 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:34:05.574416 29048 external_mini_cluster.cc:1428] Setting key 0a4a2704909efbe02d398517c6b2a6af
I20260806 04:34:05.576781 27805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.61@KRBTEST.COM" created.
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:34:05.592053 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.579319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.28.94.62:43927)
0806 04:34:05.579819 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:05.579841 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:05.580124 (+ 283us) server_negotiation.cc:407] Connection header received
0806 04:34:05.581182 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:05.581188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:05.581436 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:05.581643 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:05.582422 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.583813 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.584812 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.585222 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.585333 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.587448 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:05.587453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:05.587464 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:05.587484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:05.589193 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.589625 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.589629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.589630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.589742 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.590021 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.590025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.590027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.590386 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:05.590502 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:05.590851 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 04:34:05.591059 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260806 04:34:05.592053 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.579319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:43927 (local address 127.0.0.1:39174)
0806 04:34:05.579954 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.580000 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:05.580834 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:05.581626 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:05.581630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:05.581946 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:05.582277 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.582289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.583926 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.583929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.584708 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.584718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.585347 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.585350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.585439 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.585956 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:05.585973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:05.587316 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:05.589297 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.589306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.589313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.589529 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.589826 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.589829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.589831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.589937 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.590493 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:05.590499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:05.590632 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 04:34:05.590773 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:34:05.591122 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":101,"spinlock_wait_cycles":52992,"thread_start_us":156,"threads_started":1}
I20260806 04:34:05.594462 27805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:05.595213 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.594961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56768)
0806 04:34:05.595082 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.595150 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 04:34:05.595407 27721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
I20260806 04:34:05.596266 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.596075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56780)
0806 04:34:05.596169 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.596213 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 04:34:05.596422 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:38443: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
I20260806 04:34:05.603484 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:38443
--webserver_interface=127.28.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:34:05.645735 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } attempt: 1
I20260806 04:34:05.646715 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.646393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56782)
0806 04:34:05.646539 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.646632 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:34:05.647449 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.647206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56784)
0806 04:34:05.647310 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.647362 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:34:05.647594 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:38443: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
I20260806 04:34:05.715744 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } attempt: 2
I20260806 04:34:05.716725 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.716342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56788)
0806 04:34:05.716576 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.716637 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260806 04:34:05.717571 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.717323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56802)
0806 04:34:05.717459 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.717507 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 04:34:05.717778 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:38443: Network error: Client connection negotiation failed: client connection to 127.28.94.61:38443: connect: Connection refused (error 111)
W20260806 04:34:05.719137 27816 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:05.719480 27816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:05.719528 27816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:05.719591 27816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:05.723780 27816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:05.723841 27816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:05.723860 27816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:05.723874 27816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:05.723891 27816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:05.729419 27816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:05.730558 27816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:05.731616 27816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:05.740299 27825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:05.740299 27824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.740345 27816 server_base.cc:1034] running on GCE node
W20260806 04:34:05.740499 27827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.741142 27816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:05.742278 27816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:05.743454 27816 hybrid_clock.cc:648] HybridClock initialized: now 1785990845743421 us; error 39 us; skew 500 ppm
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:05.746234 27816 init.cc:377] Logged in from keytab as kudu/127.28.94.61@KRBTEST.COM (short username kudu)
I20260806 04:34:05.747144 27816 webserver.cc:460] Webserver started at http://127.28.94.61:38875/ using document root <none> and password file <none>
I20260806 04:34:05.747714 27816 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:05.747788 27816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:05.747988 27816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:05.750053 27816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data/instance:
uuid: "24efb6c2c88e42c7bcf054899e249720"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "95187815c0c3894f3640c8cf04692458"
server_key_iv: "4a1b8e871e07c5c747ce725a5a7de04a"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.750604 27816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal/instance:
uuid: "24efb6c2c88e42c7bcf054899e249720"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "95187815c0c3894f3640c8cf04692458"
server_key_iv: "4a1b8e871e07c5c747ce725a5a7de04a"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.754235 27816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:34:05.756529 27838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:05.757849 27816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:05.757969 27816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "24efb6c2c88e42c7bcf054899e249720"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "95187815c0c3894f3640c8cf04692458"
server_key_iv: "4a1b8e871e07c5c747ce725a5a7de04a"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.758062 27816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:05.770677 27816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:05.773712 27816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:05.773882 27816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:05.780732 27816 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.61:38443
I20260806 04:34:05.781021 27917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.61:38443 every 8 connection(s)
I20260806 04:34:05.781744 27816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
I20260806 04:34:05.784627 27919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:05.788353 27919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:05.790340 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27816
I20260806 04:34:05.790452 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-1/wal/instance
I20260806 04:34:05.790759 29048 external_mini_cluster.cc:1428] Setting key bf32523feae9a3651c6ae2e52e430e72
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.60@KRBTEST.COM" created.
I20260806 04:34:05.802357 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.790278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.28.94.62:43927)
0806 04:34:05.790412 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:05.790417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:05.791402 (+ 985us) server_negotiation.cc:407] Connection header received
0806 04:34:05.792359 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:05.792363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:05.792428 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:05.792521 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:05.793774 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.794315 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.795548 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.795792 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.795890 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.798649 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:05.798654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:05.798656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:05.798687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:05.800084 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.800629 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.800632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.800633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.800713 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.801114 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.801117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.801119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.801315 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:05.801400 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:05.802058 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 04:34:05.802206 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:34:05.803287 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.790695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:43927 (local address 127.0.0.1:39184)
0806 04:34:05.791239 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.791284 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:05.792149 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:05.792708 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:05.792715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:05.793121 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:05.793619 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.793633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.794452 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.794455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.795422 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.795434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.795967 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.795972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.796147 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.796736 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:05.796754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:05.798437 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:05.800209 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.800214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.800226 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.800516 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.800804 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.800806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.800808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.801009 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.801428 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:05.801433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:05.801707 (+ 274us) client_negotiation.cc:769] Sending connection context
0806 04:34:05.801945 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 04:34:05.802259 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260806 04:34:05.804859 27919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:05.808964 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } attempt: 3
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:34:05.815204 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.805414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56804)
0806 04:34:05.805587 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.805600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:05.805672 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:05.806213 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:05.806216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:05.806404 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:05.806516 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.806523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.807602 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.807606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.808426 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.808437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.809047 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.809051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.809150 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.809597 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:05.809611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:05.810915 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:05.813450 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.813453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.813455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.813720 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.814025 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.814027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.814029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.814126 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.814659 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:05.814663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:05.814757 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:34:05.814864 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 04:34:05.814988 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 04:34:05.815457 27928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.805519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.28.94.61:38443)
0806 04:34:05.805850 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:05.805861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:05.805882 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:34:05.805919 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:05.805924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:05.806080 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:05.806191 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:05.806745 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.807477 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.808530 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.808870 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.809023 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.811058 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:05.811067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:05.811077 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:05.811098 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:05.812815 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.813829 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.813835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.813837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.813927 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.814222 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.814226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.814227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.814505 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:05.814612 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:05.814955 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 04:34:05.815089 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":135,"threads_started":1}
I20260806 04:34:05.817373 27919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
I20260806 04:34:05.818121 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.817810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:57976)
0806 04:34:05.817979 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.818053 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 04:34:05.818291 27840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.818701 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:39349
--webserver_interface=127.28.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:34:05.819077 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.809606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56808)
0806 04:34:05.809750 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.809764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:05.809843 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:05.810139 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:05.810142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:05.810307 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:05.810420 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.810427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.811350 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.811353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.812807 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:05.812817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.813341 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.813343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:05.813422 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.813858 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:05.813871 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:05.815179 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:05.817571 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.817575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.817577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.817802 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.818098 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:05.818102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:05.818103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:05.818204 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:05.818595 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:05.818599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:05.818704 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 04:34:05.818808 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:34:05.818934 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:34:05.819118 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.818921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:57986)
0806 04:34:05.819013 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.819061 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 04:34:05.819228 27919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.819587 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.809403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.28.94.61:38443)
0806 04:34:05.809907 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:05.809915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:05.809929 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:34:05.809961 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:05.809964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:05.810006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:05.810099 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:05.810544 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.811243 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.812956 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:05.813229 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:05.813326 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:05.815349 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:05.815353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:05.815355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:05.815378 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:05.817450 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.817906 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.817909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.817910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.817991 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:05.818298 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:05.818301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:05.818303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:05.818499 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:05.818597 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:05.819322 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 04:34:05.819429 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":101,"threads_started":1}
I20260806 04:34:05.820150 27805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:05.820781 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.820530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58000)
0806 04:34:05.820674 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.820718 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 04:34:05.821426 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.821205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58008)
0806 04:34:05.821335 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.821377 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 04:34:05.821533 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.852698 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 1
I20260806 04:34:05.853506 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.853227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58012)
0806 04:34:05.853372 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.853428 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 04:34:05.854194 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.853984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58024)
0806 04:34:05.854098 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.854140 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:34:05.854305 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.868477 27919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 1
I20260806 04:34:05.869350 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.869070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58032)
0806 04:34:05.869203 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.869270 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 04:34:05.870045 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.869830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58042)
0806 04:34:05.869933 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.869983 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 04:34:05.870168 27919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.929477 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 2
I20260806 04:34:05.930338 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.930048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58050)
0806 04:34:05.930195 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.930259 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 04:34:05.930598 27931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:05.930923 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:05.930970 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:05.931032 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:34:05.931113 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.930871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58064)
0806 04:34:05.930994 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.931047 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:34:05.931262 27805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
W20260806 04:34:05.935109 27931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:05.935159 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:05.935186 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:05.935217 27931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:05.935245 27931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:05.938321 27919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 2
I20260806 04:34:05.938977 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.938733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58080)
0806 04:34:05.938858 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.938909 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:34:05.939604 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:05.939397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58094)
0806 04:34:05.939505 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:05.939550 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 04:34:05.939713 27919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:39349: Network error: Client connection negotiation failed: client connection to 127.28.94.60:39349: connect: Connection refused (error 111)
I20260806 04:34:05.940629 27931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:43927,127.28.94.61:38443,127.28.94.60:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:05.941660 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:05.942639 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:05.951078 27937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:05.951063 27938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.951246 27931 server_base.cc:1034] running on GCE node
W20260806 04:34:05.951068 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:05.951956 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:05.953091 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:05.954252 27931 hybrid_clock.cc:648] HybridClock initialized: now 1785990845954234 us; error 25 us; skew 500 ppm
Aug 06 04:34:05 dist-test-slave-ncp9 krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:05.956823 27931 init.cc:377] Logged in from keytab as kudu/127.28.94.60@KRBTEST.COM (short username kudu)
I20260806 04:34:05.957739 27931 webserver.cc:460] Webserver started at http://127.28.94.60:44437/ using document root <none> and password file <none>
I20260806 04:34:05.958294 27931 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:05.958356 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:05.958559 27931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:05.960799 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data/instance:
uuid: "1d39ecfeea7a4ec499286aa8e017054c"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "3e32512c2c17b0519537486efc462a00"
server_key_iv: "4caac471bb3f16d828defa29d40ca7e7"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.961395 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal/instance:
uuid: "1d39ecfeea7a4ec499286aa8e017054c"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "3e32512c2c17b0519537486efc462a00"
server_key_iv: "4caac471bb3f16d828defa29d40ca7e7"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.965245 27931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:34:05.967546 27954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:05.968739 27931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:05.968859 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "1d39ecfeea7a4ec499286aa8e017054c"
format_stamp: "Formatted at 2026-08-06 04:34:05 on dist-test-slave-ncp9"
server_key: "3e32512c2c17b0519537486efc462a00"
server_key_iv: "4caac471bb3f16d828defa29d40ca7e7"
server_key_version: "encryptionkey@0"
I20260806 04:34:05.969003 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:05.998667 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:06.001721 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:06.001902 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:06.008824 27931 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.60:39349
I20260806 04:34:06.009114 28028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.60:39349 every 8 connection(s)
I20260806 04:34:06.009979 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
I20260806 04:34:06.013520 28033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:06.015833 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 27931
I20260806 04:34:06.015962 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785990554567992-29048-0/minicluster-data/master-2/wal/instance
I20260806 04:34:06.016266 29048 external_mini_cluster.cc:1428] Setting key 14187b06063d9a7bbf1d6244d66c002a
I20260806 04:34:06.018431 28033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:34:06.030398 27805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 3
I20260806 04:34:06.030861 27919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } attempt: 3
I20260806 04:34:06.035337 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.020595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.28.94.62:43927)
0806 04:34:06.020721 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.020725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.021765 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 04:34:06.022793 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.022796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.022848 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.022941 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.025163 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.025883 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.027257 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.027573 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.027679 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.030358 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.030363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.030366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.030399 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.032866 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.033518 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.033522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.033524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.033630 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.034039 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.034042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.034044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.034274 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.034351 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.035047 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.035168 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:34:06.036880 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.021069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:43927 (local address 127.0.0.1:39200)
0806 04:34:06.021593 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:06.021652 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:06.022588 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:06.023019 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:06.023025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:06.023530 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:06.023798 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.023812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.025996 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.025999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.027144 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.027154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.027856 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.027861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.027979 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.028464 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:06.028486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:06.030193 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:06.033017 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.033026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.033036 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.033394 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.033748 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.033751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.033753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.033924 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.034432 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:06.034439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:06.034757 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 04:34:06.035507 (+ 750us) client_negotiation.cc:240] Negotiation successful
0806 04:34:06.035829 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260806 04:34:06.038663 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.021951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.28.94.60:39349)
0806 04:34:06.022310 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.022320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.022363 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:34:06.022582 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.022603 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.022837 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.023003 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.024366 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.025738 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.026921 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.027438 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.027604 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.030766 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.030773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.030786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.030810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.036172 (+ 5362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.036656 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.036710 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.036712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.036823 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.037161 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.037164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.037166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.037462 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.037659 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.038365 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.038506 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":161,"threads_started":1}
I20260806 04:34:06.038902 28033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:06.040664 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.031295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58122)
0806 04:34:06.031545 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:06.031559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:06.031654 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:06.031951 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:06.031954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:06.032139 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:06.032253 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.032261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.033320 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.033324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.033955 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.033966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.034424 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.034427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.034507 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.034884 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:06.034898 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:06.036303 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:06.039003 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.039006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.039008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.039236 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.039560 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.039563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.039565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.039654 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.040112 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:06.040116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:06.040218 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 04:34:06.040302 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:34:06.040487 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 04:34:06.041590 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.031325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.28.94.60:39349)
0806 04:34:06.031552 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.031560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.031624 (+ 64us) server_negotiation.cc:407] Connection header received
0806 04:34:06.031763 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.031766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.031808 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.032360 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.032471 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.033212 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.034069 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.034327 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.034411 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.036456 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.036460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.036463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.036487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.038881 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.039349 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.039352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.039354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.039441 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.039774 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.039777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.039779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.039981 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.041244 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.041317 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.041410 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260806 04:34:06.043514 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.030866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58120)
0806 04:34:06.031064 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:06.031079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:06.031158 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:06.031452 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:06.031455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:06.031613 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:06.031715 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.031721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.033731 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.033734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.034473 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.034481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.035046 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.035050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.035140 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.035622 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:06.035636 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:06.037198 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:06.041368 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.041372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.041374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.041608 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.042543 (+ 935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.042546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.042548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.042640 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.043071 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:06.043075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:06.043176 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 04:34:06.043251 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:34:06.043369 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 04:34:06.044669 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.030866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.28.94.60:39349)
0806 04:34:06.031245 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.031254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.031266 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:34:06.031298 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.031301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.031336 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.031411 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.031844 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.032617 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.034596 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.034905 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.034992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.037328 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.037332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.037334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.037361 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.041248 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.042348 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.042350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.042352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.042441 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.042750 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.042753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.042755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.042945 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.043946 (+ 1001us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.044022 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.044140 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":80,"threads_started":1}
I20260806 04:34:06.045490 27919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: Bootstrap starting.
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:34:06.048471 27919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:06.048456 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: Bootstrap starting.
I20260806 04:34:06.049417 27919 log.cc:824] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:06.050293 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.040421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.28.94.61:38443)
0806 04:34:06.040530 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.040534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.040686 (+ 152us) server_negotiation.cc:407] Connection header received
0806 04:34:06.040875 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.040878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.040966 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.041035 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.041534 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.042307 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.043388 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.043763 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.043842 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.046099 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.046102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.046105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.046131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.048268 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.048787 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.048791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.048793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.048883 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.049266 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.049269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.049271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.049539 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.049615 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.049984 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.050127 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:34:06.051057 27805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:06.051381 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.039351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:38443 (local address 127.0.0.1:56816)
0806 04:34:06.040610 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:06.040621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:06.040698 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:06.041153 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:06.041156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:06.041328 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:06.041407 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.041413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.042448 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.042452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.043244 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.043256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.043991 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.043995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.044096 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.044493 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:06.044509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:06.045936 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:06.048395 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.048399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.048401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.048668 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.049040 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.049043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.049045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.049148 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.049692 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:06.049695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:06.049793 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 04:34:06.051091 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0806 04:34:06.051230 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196}
I20260806 04:34:06.051568 27919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: No bootstrap required, opened a new log
I20260806 04:34:06.051849 27805 log.cc:824] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:06.054281 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: No bootstrap required, opened a new log
I20260806 04:34:06.054806 28033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:06.055182 27919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.055413 27919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24efb6c2c88e42c7bcf054899e249720, State: Initialized, Role: FOLLOWER
I20260806 04:34:06.055764 27919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.056216 28045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.056414 28045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.057137 27919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:06.058086 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.041728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.28.94.62:43927)
0806 04:34:06.044015 (+ 2287us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.044019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.044030 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:34:06.044061 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.044063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.044102 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.044174 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.045290 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.046006 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.047086 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.047393 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.047492 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.051546 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.051550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.051552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.051591 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.056096 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.056602 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.056606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.056607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.056696 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.057064 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.057068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.057070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.057283 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.057432 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.057816 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.057936 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2212}
I20260806 04:34:06.058763 27805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.058969 27805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04078c96a38f40b5bdedb43d07b1e0bc, State: Initialized, Role: FOLLOWER
I20260806 04:34:06.059278 27805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
W20260806 04:34:06.063374 28063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:06.063449 28063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:06.062091 27816 master_runner.cc:186] Error getting master registration for 127.28.94.62:43927: OK, instance_id { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" instance_seqno: 1785990845567349 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:34:06.065317 27816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:34:06.066013 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.055254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:39349 (local address 127.0.0.1:58126)
0806 04:34:06.055577 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:06.055591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:06.055663 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:06.057300 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:06.057303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:06.057463 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:06.057574 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.057580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.058877 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.058880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.059592 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:06.059603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.060254 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.060257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:06.060335 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.060871 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:06.060885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:06.062464 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:06.064389 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.064392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.064394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.064626 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.064911 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:06.064914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:06.064916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:06.065000 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:06.065579 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:06.065582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:06.065659 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 04:34:06.065736 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:34:06.065858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260806 04:34:06.066084 27805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:06.066782 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.055575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.28.94.60:39349)
0806 04:34:06.057044 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.057049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.057062 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:34:06.057094 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.057096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.057138 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.057676 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.057746 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.058768 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.059811 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.060142 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.060527 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.062575 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.062579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.062581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.062602 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.064286 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.064729 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.064732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.064733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.064806 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.065115 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.065118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.065120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.065475 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.066442 (+ 967us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.066522 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.066639 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384}
Aug 06 04:34:06 dist-test-slave-ncp9 krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
I20260806 04:34:06.070156 28033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: Bootstrap starting.
I20260806 04:34:06.071504 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.059588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.28.94.61:38443)
0806 04:34:06.059690 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.059694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.060318 (+ 624us) server_negotiation.cc:407] Connection header received
0806 04:34:06.060516 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.060519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.060591 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.060640 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.061488 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.062288 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.063386 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.063768 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.063847 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.068031 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.068034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.068037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.068063 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.069607 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.070075 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.070078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.070080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.070169 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.070471 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.070474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.070476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.070694 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.070880 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.071244 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.071362 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260806 04:34:06.071919 28076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:06.072002 28076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:06.072443 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.072573 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.073621 28033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:06.074532 28033 log.cc:824] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:06.076973 28033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: No bootstrap required, opened a new log
I20260806 04:34:06.077616 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:34:06.075778 27697 master_runner.cc:186] Error getting master registration for 127.28.94.60:39349: OK, instance_id { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" instance_seqno: 1785990846006885 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:34:06.080319 27697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:34:06.080705 27761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "04078c96a38f40b5bdedb43d07b1e0bc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39216
I20260806 04:34:06.080978 27761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:34:06.081108 27761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:06.082002 28033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.082355 28033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d39ecfeea7a4ec499286aa8e017054c, State: Initialized, Role: FOLLOWER
I20260806 04:34:06.082345 27761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.082760 28033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.083240 27761 leader_election.cc:288] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 24efb6c2c88e42c7bcf054899e249720 (127.28.94.61:38443), 1d39ecfeea7a4ec499286aa8e017054c (127.28.94.60:39349)
I20260806 04:34:06.083330 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.083480 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.084375 28033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:06.085649 27991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d39ecfeea7a4ec499286aa8e017054c"
I20260806 04:34:06.085860 27991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:06.087190 27991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04078c96a38f40b5bdedb43d07b1e0bc in term 1.
I20260806 04:34:06.087800 27879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24efb6c2c88e42c7bcf054899e249720"
I20260806 04:34:06.087838 27721 leader_election.cc:302] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04078c96a38f40b5bdedb43d07b1e0bc, 1d39ecfeea7a4ec499286aa8e017054c; no voters:
I20260806 04:34:06.088025 27879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:06.088732 28053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:34:06.088960 28053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [term 1 LEADER]: Becoming Leader. State: Replica: 04078c96a38f40b5bdedb43d07b1e0bc, State: Running, Role: LEADER
I20260806 04:34:06.089296 28053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } }
I20260806 04:34:06.089349 27879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04078c96a38f40b5bdedb43d07b1e0bc in term 1.
I20260806 04:34:06.091212 28053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04078c96a38f40b5bdedb43d07b1e0bc. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.091336 28053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: This master's current role is: LEADER
W20260806 04:34:06.091586 28101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:06.091658 28101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:06.102289 28106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:34:06.103345 28106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:34:06.104543 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.087682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.28.94.62:43927)
0806 04:34:06.087797 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.087802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.091746 (+ 3944us) server_negotiation.cc:407] Connection header received
0806 04:34:06.091920 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.091923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.091965 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.092020 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.092963 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.093726 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.097202 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.097531 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.097603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.100701 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.100706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.100709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.100739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.102422 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.102883 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.102887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.102889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.102977 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.103282 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.103286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.103287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.103508 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.103882 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.104263 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.104385 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:34:06.108217 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.099742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.28.94.61:38443)
0806 04:34:06.099875 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.099879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.100162 (+ 283us) server_negotiation.cc:407] Connection header received
0806 04:34:06.100286 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.100290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.100339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.100429 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.101100 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.101808 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.102858 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.103153 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.103262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.104690 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.104694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.104696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.104722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.106436 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.106891 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.106894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.106896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.106987 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.107289 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.107292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.107294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.107515 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.107608 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.107945 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.108068 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:34:06.114434 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:06.100285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.28.94.60:39349)
0806 04:34:06.100398 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.100402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.105055 (+ 4653us) server_negotiation.cc:407] Connection header received
0806 04:34:06.105231 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.105235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.105287 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.105341 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.106056 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.106746 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.107734 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:06.108051 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:06.108126 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:06.110825 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:06.110829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:06.110832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:06.110859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:06.112500 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.113031 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.113035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.113037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.113127 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:06.113446 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:06.113450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:06.113452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:06.113668 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:06.113794 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:06.114163 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 04:34:06.114282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:34:06.119104 27879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [term 1 FOLLOWER]: Refusing update from remote peer 04078c96a38f40b5bdedb43d07b1e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:34:06.119829 28102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:34:06.121160 27991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [term 1 FOLLOWER]: Refusing update from remote peer 04078c96a38f40b5bdedb43d07b1e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:34:06.123842 28045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04078c96a38f40b5bdedb43d07b1e0bc. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.123950 28045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.124755 28102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:34:06.127394 28045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.127496 28045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.128156 28106 catalog_manager.cc:1347] Generated new cluster ID: 78e8ac43d36b4bb1a652509574ca47ef
I20260806 04:34:06.128212 28106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:34:06.128808 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04078c96a38f40b5bdedb43d07b1e0bc. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.128947 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.132392 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.132511 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:06.137091 28111 mvcc.cc:205] Tried to move back new op lower bound from 7315418505693093888 to 7315418505585668096. Current Snapshot: MvccSnapshot[applied={T|T < 7315418505693093888}]
I20260806 04:34:06.137344 28110 mvcc.cc:205] Tried to move back new op lower bound from 7315418505693093888 to 7315418505585668096. Current Snapshot: MvccSnapshot[applied={T|T < 7315418505693093888 or (T in {7315418505693093888})}]
I20260806 04:34:06.138149 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.138276 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:06.138379 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04078c96a38f40b5bdedb43d07b1e0bc" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 43927 } } peers { permanent_uuid: "24efb6c2c88e42c7bcf054899e249720" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 38443 } } peers { permanent_uuid: "1d39ecfeea7a4ec499286aa8e017054c" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 39349 } } }
I20260806 04:34:06.138456 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:06.142828 28120 mvcc.cc:205] Tried to move back new op lower bound from 7315418505693093888 to 7315418505585668096. Current Snapshot: MvccSnapshot[applied={T|T < 7315418505693093888}]
I20260806 04:34:06.165017 28106 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:34:06.166347 28106 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:34:06.183607 28106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04078c96a38f40b5bdedb43d07b1e0bc: Generated new TSK 0
I20260806 04:34:06.184541 28106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:34:06.216477 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.213470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.28.94.61:38443)
0806 04:34:06.213635 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.213640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.215362 (+ 1722us) server_negotiation.cc:407] Connection header received
0806 04:34:06.215582 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.215586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.215640 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.215692 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.216331 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56856: BlockingRecv error: recv got EOF from 127.0.0.1:56856 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 04:34:06.217999 28040 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.214284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.28.94.60:39349)
0806 04:34:06.214399 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.214403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.217067 (+ 2664us) server_negotiation.cc:407] Connection header received
0806 04:34:06.217246 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.217250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.217301 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.217349 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.217870 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 04:34:06.219373 27811 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.214921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.28.94.62:43927)
0806 04:34:06.215061 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.215066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.218478 (+ 3412us) server_negotiation.cc:407] Connection header received
0806 04:34:06.218669 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.218672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.218720 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.218768 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.219252 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39226: BlockingRecv error: recv got EOF from 127.0.0.1:39226 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:34:06.223835 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.222267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.28.94.61:38443)
0806 04:34:06.222371 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.222375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.222878 (+ 503us) server_negotiation.cc:407] Connection header received
0806 04:34:06.223097 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.223100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.223180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.223228 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.223733 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56868: BlockingRecv error: recv got EOF from 127.0.0.1:56868 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 04:34:06.228688 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.227336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.28.94.61:38443)
0806 04:34:06.227431 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.227435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.227875 (+ 440us) server_negotiation.cc:407] Connection header received
0806 04:34:06.228038 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.228041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.228088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.228133 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.228593 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56876: BlockingRecv error: recv got EOF from 127.0.0.1:56876 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 04:34:06.234418 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.232935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.28.94.61:38443)
0806 04:34:06.233042 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.233046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.233530 (+ 484us) server_negotiation.cc:407] Connection header received
0806 04:34:06.233723 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.233727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.233776 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.233823 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.234314 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56880: BlockingRecv error: recv got EOF from 127.0.0.1:56880 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 04:34:06.242323 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.240655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.28.94.61:38443)
0806 04:34:06.240791 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.240794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.241350 (+ 556us) server_negotiation.cc:407] Connection header received
0806 04:34:06.241563 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.241566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.241620 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.241673 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.242220 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56888: BlockingRecv error: recv got EOF from 127.0.0.1:56888 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 04:34:06.254729 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.252842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.28.94.61:38443)
0806 04:34:06.252992 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.252997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.253673 (+ 676us) server_negotiation.cc:407] Connection header received
0806 04:34:06.253903 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.253907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.253966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.254022 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.254605 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56890: BlockingRecv error: recv got EOF from 127.0.0.1:56890 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 04:34:06.275352 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.273619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.28.94.61:38443)
0806 04:34:06.273742 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.273747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.274352 (+ 605us) server_negotiation.cc:407] Connection header received
0806 04:34:06.274565 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.274569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.274625 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.274675 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.275227 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56904: BlockingRecv error: recv got EOF from 127.0.0.1:56904 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 04:34:06.311112 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.309906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.28.94.61:38443)
0806 04:34:06.310045 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.310050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.310256 (+ 206us) server_negotiation.cc:407] Connection header received
0806 04:34:06.310390 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.310393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.310443 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.310512 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.310974 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56920: BlockingRecv error: recv got EOF from 127.0.0.1:56920 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 04:34:06.378531 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.377231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.28.94.61:38443)
0806 04:34:06.377432 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.377462 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.377544 (+ 82us) server_negotiation.cc:407] Connection header received
0806 04:34:06.377745 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.377748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.377804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.377885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.378406 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56930: BlockingRecv error: recv got EOF from 127.0.0.1:56930 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 04:34:06.509665 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.508639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.28.94.61:38443)
0806 04:34:06.508770 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.508774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.508882 (+ 108us) server_negotiation.cc:407] Connection header received
0806 04:34:06.509045 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.509047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.509101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.509183 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.509559 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56934: BlockingRecv error: recv got EOF from 127.0.0.1:56934 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 04:34:06.768841 27929 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:06.767824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.28.94.61:38443)
0806 04:34:06.767938 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:06.767942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:06.768004 (+ 62us) server_negotiation.cc:407] Connection header received
0806 04:34:06.768164 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:06.768167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:06.768220 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:06.768286 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:06.768720 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56940: BlockingRecv error: recv got EOF from 127.0.0.1:56940 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:34:07.065109 28063 catalog_manager.cc:1259] Loaded cluster ID: 78e8ac43d36b4bb1a652509574ca47ef
I20260806 04:34:07.065194 28063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: loading cluster ID for follower catalog manager: success
I20260806 04:34:07.066973 28063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: acquiring CA information for follower catalog manager: success
I20260806 04:34:07.068235 28063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24efb6c2c88e42c7bcf054899e249720: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:34:07.092936 28101 catalog_manager.cc:1259] Loaded cluster ID: 78e8ac43d36b4bb1a652509574ca47ef
I20260806 04:34:07.093008 28101 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: loading cluster ID for follower catalog manager: success
I20260806 04:34:07.094667 28101 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: acquiring CA information for follower catalog manager: success
I20260806 04:34:07.095795 28101 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d39ecfeea7a4ec499286aa8e017054c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:34:07.286978 28211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:07.283000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.28.94.61:38443)
0806 04:34:07.283265 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:07.283268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:07.283285 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:34:07.283425 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:07.283427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:07.283558 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:07.283660 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:07.283918 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.284512 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.285481 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.285726 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.285823 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:07.286044 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:07.286569 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:07.286671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:07.286728 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 04:34:07.286785 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
I20260806 04:34:07.293332 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:07.290118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.28.94.60:39349)
0806 04:34:07.290334 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:07.290338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:07.290351 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:34:07.290429 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:07.290432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:07.290540 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:07.290641 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:07.290793 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.291296 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.292024 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.292261 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.292336 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:07.292548 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:07.292979 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:07.293106 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:07.293162 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 04:34:07.293214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260806 04:34:07.300213 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:07.296504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.28.94.60:39349)
0806 04:34:07.296637 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:07.296641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:07.296936 (+ 295us) server_negotiation.cc:407] Connection header received
0806 04:34:07.297129 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:07.297132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:07.297177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:07.297234 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:07.297472 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.298256 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299114 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.299370 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299473 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:07.299740 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:07.299821 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:07.299867 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:07.300039 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 04:34:07.300092 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:34:07.300223 28211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:07.296081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.28.94.61:38443)
0806 04:34:07.296199 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:07.296203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:07.296585 (+ 382us) server_negotiation.cc:407] Connection header received
0806 04:34:07.296761 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:07.296764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:07.296814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:07.296872 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:07.297337 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.297986 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299110 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.299394 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299460 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:07.299774 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:07.299841 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:07.299881 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:07.300033 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 04:34:07.300097 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:34:07.300869 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:07.296147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.28.94.62:43927)
0806 04:34:07.296369 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:07.296373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:07.296660 (+ 287us) server_negotiation.cc:407] Connection header received
0806 04:34:07.296822 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:07.296825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:07.297009 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:07.297154 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:07.297375 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.298207 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299267 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:07.299594 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:07.299687 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:07.299992 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:07.300304 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:07.300641 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:07.300678 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 04:34:07.300737 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20260806 04:34:07.304039 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27697
I20260806 04:34:07.321177 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27816
I20260806 04:34:07.332667 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 27931
2026-08-06T04:34:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3196 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:34:07 dist-test-slave-ncp9 krb5kdc[28231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:34:07 dist-test-slave-ncp9 krb5kdc[28231](info): set up 2 sockets
Aug 06 04:34:07 dist-test-slave-ncp9 krb5kdc[28231](info): commencing operation
krb5kdc: starting...
W20260806 04:34:08.440830 29048 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/test-user.keytab.
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.94.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
Entry for principal kudu/127.28.94.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab.
I20260806 04:34:08.563472 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33101
--webserver_interface=127.28.94.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:34:08.674757 28246 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:08.675038 28246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:08.675061 28246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:08.675107 28246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:34:08.679116 28246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:08.679176 28246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:08.679195 28246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:08.679210 28246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:08.679224 28246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:08.684453 28246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.62.keytab
--principal=kudu/127.28.94.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.62:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.94.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:08.685560 28246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:08.686587 28246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:08.694896 28253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:08.694902 28257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:08.694932 28255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:08.695127 28246 server_base.cc:1034] running on GCE node
I20260806 04:34:08.695748 28246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:08.696841 28246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:08.698036 28246 hybrid_clock.cc:648] HybridClock initialized: now 1785990848698014 us; error 32 us; skew 500 ppm
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:08.700984 28246 init.cc:377] Logged in from keytab as kudu/127.28.94.62@KRBTEST.COM (short username kudu)
I20260806 04:34:08.701947 28246 webserver.cc:460] Webserver started at http://127.28.94.62:37297/ using document root <none> and password file <none>
I20260806 04:34:08.702512 28246 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:08.702574 28246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:08.702780 28246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:08.704830 28246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data/instance:
uuid: "ca759490b047427bbf430d04c0d3ea05"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "7e984f0a8356f1d21bf080ca564e0b48"
server_key_iv: "8b58d322d41d16b362b26149b64e41a4"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.705456 28246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance:
uuid: "ca759490b047427bbf430d04c0d3ea05"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "7e984f0a8356f1d21bf080ca564e0b48"
server_key_iv: "8b58d322d41d16b362b26149b64e41a4"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.709224 28246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:34:08.711570 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:08.712765 28246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:08.712886 28246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal
uuid: "ca759490b047427bbf430d04c0d3ea05"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "7e984f0a8356f1d21bf080ca564e0b48"
server_key_iv: "8b58d322d41d16b362b26149b64e41a4"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.713018 28246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:08.730638 28246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:08.733419 28246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:08.733596 28246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:08.740666 28246 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.62:33101
I20260806 04:34:08.740962 28346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.62:33101 every 8 connection(s)
I20260806 04:34:08.741662 28246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/data/info.pb
I20260806 04:34:08.744452 28348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:08.748169 28348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:08.749688 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 28246
I20260806 04:34:08.749805 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-0/wal/instance
I20260806 04:34:08.750097 29048 external_mini_cluster.cc:1428] Setting key 54b26520a97cdbf831daaae07c642162
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.94.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.61@KRBTEST.COM" created.
I20260806 04:34:08.763103 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.750366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33101 (local address 127.0.0.1:41230)
0806 04:34:08.750847 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.750906 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:08.751913 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:08.752709 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:08.752713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:08.753082 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:08.753332 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.753345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.754840 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.754842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.755619 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.755628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.756340 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.756345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.756453 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.757043 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:08.757064 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:08.758701 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:08.760399 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.760410 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.760420 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.760663 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.760981 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.760983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.760985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.761094 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.761688 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:08.761694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:08.761818 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 04:34:08.761957 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 04:34:08.762312 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":95,"thread_start_us":104,"threads_started":1}
I20260806 04:34:08.763105 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.751059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.28.94.62:33101)
0806 04:34:08.751311 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:08.751333 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:08.751368 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:34:08.752249 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:08.752255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:08.752495 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:08.752698 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:08.753474 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.754733 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.755739 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.756175 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.756297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.758843 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:08.758849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:08.758858 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:08.758879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:08.760295 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.760762 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.760766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.760768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.760871 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.761177 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.761179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.761180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.761551 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:08.761671 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:08.762039 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 04:34:08.762228 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260806 04:34:08.765794 28348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:08.766530 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.766277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50550)
0806 04:34:08.766407 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.766475 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 04:34:08.766794 28274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
I20260806 04:34:08.767661 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.767418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50554)
0806 04:34:08.767551 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.767601 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 04:34:08.767993 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41319: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
Entry for principal kudu/127.28.94.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab.
I20260806 04:34:08.777490 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:41319
--webserver_interface=127.28.94.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:34:08.817189 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } attempt: 1
I20260806 04:34:08.818089 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.817780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50564)
0806 04:34:08.817938 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.818004 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:34:08.818814 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.818601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50580)
0806 04:34:08.818710 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.818755 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 04:34:08.818959 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41319: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
I20260806 04:34:08.887125 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } attempt: 2
W20260806 04:34:08.887253 28358 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:08.887569 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:08.887614 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:08.887682 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:34:08.887987 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.887691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50584)
0806 04:34:08.887834 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.887896 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 04:34:08.888759 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.888549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50596)
0806 04:34:08.888658 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.888704 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:34:08.888883 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.61:41319: Network error: Client connection negotiation failed: client connection to 127.28.94.61:41319: connect: Connection refused (error 111)
W20260806 04:34:08.892010 28358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:08.892060 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:08.892086 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:08.892117 28358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:08.892144 28358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:08.897274 28358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.61.keytab
--principal=kudu/127.28.94.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.61:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.94.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:08.898244 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:08.899191 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:34:08.907438 28367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:08.907450 28365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:08.907503 28358 server_base.cc:1034] running on GCE node
W20260806 04:34:08.907450 28364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:08.908185 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:08.909243 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:08.910404 28358 hybrid_clock.cc:648] HybridClock initialized: now 1785990848910383 us; error 32 us; skew 500 ppm
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:08.913092 28358 init.cc:377] Logged in from keytab as kudu/127.28.94.61@KRBTEST.COM (short username kudu)
I20260806 04:34:08.913916 28358 webserver.cc:460] Webserver started at http://127.28.94.61:37449/ using document root <none> and password file <none>
I20260806 04:34:08.914511 28358 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:08.914592 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:08.914827 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:08.916802 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data/instance:
uuid: "eed994138812448fab5da49cea7bf73b"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "61cbe8c16f5d9d867a172796a6bd1f66"
server_key_iv: "939251d44c8535846cbc07823a599da6"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.917361 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal/instance:
uuid: "eed994138812448fab5da49cea7bf73b"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "61cbe8c16f5d9d867a172796a6bd1f66"
server_key_iv: "939251d44c8535846cbc07823a599da6"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.920797 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:34:08.923120 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:08.924263 28358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:34:08.924377 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal
uuid: "eed994138812448fab5da49cea7bf73b"
format_stamp: "Formatted at 2026-08-06 04:34:08 on dist-test-slave-ncp9"
server_key: "61cbe8c16f5d9d867a172796a6bd1f66"
server_key_iv: "939251d44c8535846cbc07823a599da6"
server_key_version: "encryptionkey@0"
I20260806 04:34:08.924466 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:08.948954 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:08.952294 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:08.952507 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:08.960250 28358 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.61:41319
I20260806 04:34:08.960510 28443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.61:41319 every 8 connection(s)
I20260806 04:34:08.961266 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/data/info.pb
I20260806 04:34:08.963196 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 28358
I20260806 04:34:08.963321 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-1/wal/instance
I20260806 04:34:08.963639 29048 external_mini_cluster.cc:1428] Setting key 4be1c2eb4577b7ac503d0dbc8c97354c
I20260806 04:34:08.964387 28444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:08.968384 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.94.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.94.60@KRBTEST.COM" created.
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:34:08.980073 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } attempt: 3
I20260806 04:34:08.982158 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.970232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.28.94.62:33101)
0806 04:34:08.970385 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:08.970390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:08.971200 (+ 810us) server_negotiation.cc:407] Connection header received
0806 04:34:08.972129 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:08.972132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:08.972186 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:08.972266 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:08.973450 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.974016 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.975441 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.975701 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.975793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.978243 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:08.978247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:08.978250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:08.978280 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:08.979670 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.980178 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.980182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.980184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.980283 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.980938 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.980942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.980944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.981188 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:08.981262 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:08.981879 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 04:34:08.982000 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:34:08.983083 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.970566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33101 (local address 127.0.0.1:41234)
0806 04:34:08.971030 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.971081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:08.971933 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:08.972402 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:08.972408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:08.972801 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:08.973317 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.973330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.974142 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.974144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.975301 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.975313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.975859 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.975863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.976083 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.976697 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:08.976713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:08.978075 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:08.979785 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.979790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.979801 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.980063 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.980391 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.980394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.980396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.980806 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.981343 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:08.981348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:08.981640 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 04:34:08.981801 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 04:34:08.982114 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":4,"thread_start_us":112,"threads_started":1}
I20260806 04:34:08.984491 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
Aug 06 04:34:08 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
Entry for principal kudu/127.28.94.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab.
I20260806 04:34:08.991062 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.980489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50612)
0806 04:34:08.980609 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.980623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:08.980692 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:08.981399 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:08.981403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:08.981581 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:08.981702 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.981709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.983689 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.983692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.984797 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.984808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.985545 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.985549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.985680 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.986088 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:08.986102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:08.987431 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:08.989432 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.989435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.989437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.989676 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.989999 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.990002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.990004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.990101 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.990564 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:08.990567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:08.990668 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 04:34:08.990769 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 04:34:08.990894 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 04:34:08.991659 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.980607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.28.94.61:41319)
0806 04:34:08.980984 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:08.980996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:08.981018 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:34:08.981056 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:08.981062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:08.981257 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:08.981384 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:08.982336 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.983602 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.984955 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.985428 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.985538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.987594 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:08.987606 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:08.987620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:08.987647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:08.989330 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.989782 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.989787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.989788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.989876 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.990185 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.990188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.990190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.990450 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:08.990572 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:08.991409 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 04:34:08.991525 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":3840,"thread_start_us":149,"threads_started":1}
I20260806 04:34:08.991761 29048 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4FkAn1/build/debug/bin/kudu
/tmp/dist-test-task4FkAn1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:42919
--webserver_interface=127.28.94.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.94.20:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:34:08.992594 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.984975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50622)
0806 04:34:08.985221 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.985233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:08.985304 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:08.985494 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:08.985497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:08.985641 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:08.985744 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.985750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.986423 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.986426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.987183 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:08.987195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.987579 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.987582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:08.987671 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.988033 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:08.988045 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:08.989125 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:08.991196 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.991198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.991199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.991414 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.991665 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:08.991668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:08.991670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:08.991742 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:08.992132 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:08.992135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:08.992226 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 04:34:08.992328 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 04:34:08.992438 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":1,"spinlock_wait_cycles":1408}
I20260806 04:34:08.992666 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.984985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.28.94.61:41319)
0806 04:34:08.985256 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:08.985264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:08.985277 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:34:08.985368 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:08.985371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:08.985407 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:08.985472 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:08.985837 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.986332 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.987275 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:08.987489 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:08.987576 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:08.989241 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:08.989244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:08.989247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:08.989271 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:08.991088 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.991505 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.991507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.991509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.991581 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:08.991820 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:08.991824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:08.991825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:08.992031 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:08.992124 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:08.992388 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 04:34:08.992497 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260806 04:34:08.993462 28444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:08.993884 28348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:08.994134 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.993856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53772)
0806 04:34:08.994001 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.994075 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 04:34:08.994311 28380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
I20260806 04:34:08.994589 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.994330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53778)
0806 04:34:08.994454 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.994519 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 04:34:08.995292 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.995112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53796)
0806 04:34:08.995198 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.995243 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 04:34:08.995431 28444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
I20260806 04:34:08.995620 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:08.995002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53792)
0806 04:34:08.995512 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:08.995560 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260806 04:34:08.995739 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
I20260806 04:34:09.026916 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 1
I20260806 04:34:09.027820 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.027452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53810)
0806 04:34:09.027658 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.027733 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 04:34:09.028653 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.028344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53826)
0806 04:34:09.028478 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.028552 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 04:34:09.028847 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
I20260806 04:34:09.044675 28444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 1
I20260806 04:34:09.045372 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.045116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53836)
0806 04:34:09.045245 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.045299 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:34:09.046164 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.045952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53846)
0806 04:34:09.046064 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.046106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 04:34:09.046270 28444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
W20260806 04:34:09.103767 28454 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:34:09.104022 28454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:34:09.104055 28454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:34:09.104116 28454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:34:09.104023 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 2
I20260806 04:34:09.104868 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.104542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53860)
0806 04:34:09.104722 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.104788 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 04:34:09.105592 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.105397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53868)
0806 04:34:09.105479 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.105529 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 04:34:09.105717 28348 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
W20260806 04:34:09.108426 28454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:34:09.108484 28454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:34:09.108512 28454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:34:09.108542 28454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:34:09.108569 28454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:34:09.113669 28454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.94.20:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.94.62:33101,127.28.94.61:41319,127.28.94.60:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/krb5kdc/kudu_127.28.94.60.keytab
--principal=kudu/127.28.94.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.94.60:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.94.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:34:09.114423 28444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 2
I20260806 04:34:09.114696 28454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:34:09.115186 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.114904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53870)
0806 04:34:09.115054 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.115112 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:34:09.115770 28454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:34:09.115984 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.115724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53876)
0806 04:34:09.115861 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.115912 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:34:09.116104 28444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.94.60:42919: Network error: Client connection negotiation failed: client connection to 127.28.94.60:42919: connect: Connection refused (error 111)
W20260806 04:34:09.124102 28461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:09.124109 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:34:09.124490 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:34:09.125253 28454 server_base.cc:1034] running on GCE node
I20260806 04:34:09.125830 28454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:34:09.127075 28454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:34:09.128245 28454 hybrid_clock.cc:648] HybridClock initialized: now 1785990849128228 us; error 23 us; skew 500 ppm
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990849, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:34:09.131018 28454 init.cc:377] Logged in from keytab as kudu/127.28.94.60@KRBTEST.COM (short username kudu)
I20260806 04:34:09.131881 28454 webserver.cc:460] Webserver started at http://127.28.94.60:41913/ using document root <none> and password file <none>
I20260806 04:34:09.132427 28454 fs_manager.cc:359] Metadata directory not provided
I20260806 04:34:09.132505 28454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:34:09.132731 28454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:34:09.134773 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data/instance:
uuid: "0b83d7d3e63b4c058a88e961807acfb4"
format_stamp: "Formatted at 2026-08-06 04:34:09 on dist-test-slave-ncp9"
server_key: "014185f023496fb3792b8c6c27c8f086"
server_key_iv: "ef21d3e1060fe1f7477d57c34fe19e70"
server_key_version: "encryptionkey@0"
I20260806 04:34:09.135344 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal/instance:
uuid: "0b83d7d3e63b4c058a88e961807acfb4"
format_stamp: "Formatted at 2026-08-06 04:34:09 on dist-test-slave-ncp9"
server_key: "014185f023496fb3792b8c6c27c8f086"
server_key_iv: "ef21d3e1060fe1f7477d57c34fe19e70"
server_key_version: "encryptionkey@0"
I20260806 04:34:09.139082 28454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:34:09.141526 28483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:34:09.142756 28454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:34:09.142881 28454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data,/tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal
uuid: "0b83d7d3e63b4c058a88e961807acfb4"
format_stamp: "Formatted at 2026-08-06 04:34:09 on dist-test-slave-ncp9"
server_key: "014185f023496fb3792b8c6c27c8f086"
server_key_iv: "ef21d3e1060fe1f7477d57c34fe19e70"
server_key_version: "encryptionkey@0"
I20260806 04:34:09.142992 28454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:34:09.161063 28454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:34:09.164260 28454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:34:09.164458 28454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:34:09.171658 28454 rpc_server.cc:307] RPC server started. Bound to: 127.28.94.60:42919
I20260806 04:34:09.171965 28559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.94.60:42919 every 8 connection(s)
I20260806 04:34:09.172660 28454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/data/info.pb
I20260806 04:34:09.175426 28560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:34:09.179152 29048 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4FkAn1/build/debug/bin/kudu as pid 28454
I20260806 04:34:09.179286 29048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4FkAn1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785990554567992-29048-0/minicluster-data/master-2/wal/instance
I20260806 04:34:09.179109 28560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:09.179570 29048 external_mini_cluster.cc:1428] Setting key 2b6bafda096345995301a6460de2daac
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990849, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:34:09.191285 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.181092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.28.94.62:33101)
0806 04:34:09.181208 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.181213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.181598 (+ 385us) server_negotiation.cc:407] Connection header received
0806 04:34:09.182434 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.182437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.182495 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.182582 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.183406 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.183899 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.184851 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.185123 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.185214 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.187663 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.187667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.187670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.187700 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.189171 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.189734 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.189737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.189738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.189815 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.190132 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.190136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.190138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.190357 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.190456 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.191001 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.191122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:34:09.192157 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.181039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.62:33101 (local address 127.0.0.1:41242)
0806 04:34:09.181442 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.181491 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:09.182187 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:09.182658 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:09.182664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:09.183014 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:09.183264 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.183277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.184006 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.184009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.184734 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.184743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.185272 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.185276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.185447 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.185978 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:09.185999 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:09.187490 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:09.189302 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.189309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.189319 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.189637 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.189906 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.189909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.189911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.190048 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.190457 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:09.190463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:09.190747 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 04:34:09.190906 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:34:09.191188 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260806 04:34:09.192157 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.180865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.28.94.60:42919)
0806 04:34:09.181386 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.181403 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.181441 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:34:09.182183 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.182219 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.182460 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.182614 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.183393 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.184206 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.185056 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.185398 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.185497 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.188053 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.188059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.188072 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.188097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.189803 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.190197 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.190201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.190203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.190269 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.190530 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.190533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.190535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.190802 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.190988 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.191266 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.191397 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":128,"threads_started":1}
I20260806 04:34:09.193914 28560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990849, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.62@KRBTEST.COM
I20260806 04:34:09.201848 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.194328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.61:41319 (local address 127.0.0.1:50638)
0806 04:34:09.194422 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.194435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:09.194501 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:09.194922 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:09.194925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:09.195073 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:09.195155 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.195160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.195868 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.195872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.196383 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.196391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.196843 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.196847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.196981 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.197294 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:09.197303 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:09.198487 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:09.200361 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.200364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.200365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.200542 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.200803 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.200806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.200808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.200931 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.201372 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:09.201375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:09.201472 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 04:34:09.201574 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 04:34:09.201692 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 04:34:09.201929 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.194608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.28.94.61:41319)
0806 04:34:09.194731 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.194734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.194745 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:34:09.194781 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.194783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.194822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.194889 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.195269 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.195769 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.196497 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.196745 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.196831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.198638 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.198641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.198643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.198668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.200257 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.200638 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.200642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.200643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.200721 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.201011 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.201014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.201015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.201212 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.201323 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.201663 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.201781 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:34:09.202260 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.194281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.28.94.62:33101)
0806 04:34:09.194385 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.194389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.194403 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:34:09.194476 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.194478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.194521 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.194618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.195165 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.195625 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.196325 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.196521 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.196610 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.199176 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.199180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.199183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.199208 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.200529 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.200939 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.200943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.200944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.201031 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.201264 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.201267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.201269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.201509 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.201704 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.201859 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.201974 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:34:09.202860 28560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:34:09.204865 28348 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 3
I20260806 04:34:09.207264 28444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } attempt: 3
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990849, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.60@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.94.61@KRBTEST.COM
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.62@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:34:09.211735 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.203296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53890)
0806 04:34:09.203385 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.203398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:09.203475 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:09.203707 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:09.203709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:09.203845 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:09.203903 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.203907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.204553 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.204556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.205162 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.205172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.205769 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.205772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.205841 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.206351 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:09.206363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:09.208280 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:09.209993 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.209995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.209997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.210265 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.210578 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.210580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.210582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.210677 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.211236 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:09.211239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:09.211327 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 04:34:09.211426 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:34:09.211564 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":2}
I20260806 04:34:09.212158 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.203467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.28.94.60:42919)
0806 04:34:09.203552 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.203556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.203566 (+ 10us) server_negotiation.cc:407] Connection header received
0806 04:34:09.203592 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.203594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.203625 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.203683 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.204001 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.204463 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.205281 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.205590 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.205677 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.208402 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.208405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.208408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.208429 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.209880 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.210402 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.210405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.210407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.210490 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.210773 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.210776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.210778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.210958 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.211037 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.211923 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.212028 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:34:09.212328 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.203330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.28.94.61:41319)
0806 04:34:09.203421 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.203425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.203492 (+ 67us) server_negotiation.cc:407] Connection header received
0806 04:34:09.203615 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.203618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.203657 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.203716 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.204278 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.204729 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.205667 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.205970 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.206066 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.208485 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.208489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.208491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.208513 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.210350 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.210845 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.210849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.210851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.210931 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.211208 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.211211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.211213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.211403 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.211599 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.212073 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.212183 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Aug 06 04:34:09 dist-test-slave-ncp9 krb5kdc[28231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785990848, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.94.61@KRBTEST.COM for kudu/127.28.94.60@KRBTEST.COM
I20260806 04:34:09.214911 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.205239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53892)
0806 04:34:09.205384 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.205395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:09.205456 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:09.205879 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:09.205882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:09.206044 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:09.206131 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.206139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.207608 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.207612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.208379 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.208389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.208980 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.208984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.209070 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.209512 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:09.209528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:09.210875 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:09.213336 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.213339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.213341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.213582 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.213893 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.213896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.213898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.213986 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.214446 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:09.214449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:09.214541 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 04:34:09.214628 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:34:09.214754 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 04:34:09.214942 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.205293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.28.94.60:42919)
0806 04:34:09.205691 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.205700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.205712 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:34:09.205739 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.205741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.205772 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.205840 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.206242 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.207477 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.208507 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.208832 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.208966 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.211019 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.211023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.211025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.211046 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.213235 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.213697 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.213701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.213703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.213790 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.214101 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.214104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.214106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.214331 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.214446 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.214693 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.214790 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":332,"thread_start_us":125,"threads_started":1}
I20260806 04:34:09.215698 28560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: Bootstrap starting.
I20260806 04:34:09.217792 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.207581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.94.60:42919 (local address 127.0.0.1:53906)
0806 04:34:09.207729 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:34:09.207741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:34:09.207812 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:34:09.208223 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:34:09.208226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:34:09.208382 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:34:09.208474 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.208480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.210577 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.210580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.211568 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:34:09.211578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.212415 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.212419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:34:09.212497 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.212953 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:34:09.212967 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:34:09.214280 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:34:09.216182 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.216185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.216187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.216435 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.216697 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:34:09.216700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:34:09.216702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:34:09.216791 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:34:09.217282 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:34:09.217285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:34:09.217382 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 04:34:09.217490 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:34:09.217645 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 04:34:09.217847 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.207664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.28.94.60:42919)
0806 04:34:09.207985 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.207993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.208005 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:34:09.208047 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.208049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.208087 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.208571 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.209685 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.210463 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.211698 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.212288 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.213713 (+ 1425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.214413 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.214416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.214419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.214442 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.216074 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.216516 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.216519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.216521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.216606 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.216879 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.216882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.216884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.217172 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.217265 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.217593 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.217702 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260806 04:34:09.218195 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: Bootstrap starting.
I20260806 04:34:09.218595 28560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:09.219449 28560 log.cc:824] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:09.220343 28348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:09.220729 28444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: Bootstrap starting.
I20260806 04:34:09.221227 28348 log.cc:824] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:09.221968 28560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: No bootstrap required, opened a new log
I20260806 04:34:09.223162 29048 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:34:09.223174 28444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: Neither blocks nor log segments found. Creating new log.
I20260806 04:34:09.223949 28444 log.cc:824] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:34:09.224223 28307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ca759490b047427bbf430d04c0d3ea05"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254
I20260806 04:34:09.225512 28348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: No bootstrap required, opened a new log
I20260806 04:34:09.226083 28444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: No bootstrap required, opened a new log
I20260806 04:34:09.226553 28560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.226788 28560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b83d7d3e63b4c058a88e961807acfb4, State: Initialized, Role: FOLLOWER
I20260806 04:34:09.227231 28560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.227806 28575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.228109 28575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.228582 28560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:09.228508 28348 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.228731 28348 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca759490b047427bbf430d04c0d3ea05, State: Initialized, Role: FOLLOWER
I20260806 04:34:09.229049 28348 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.229576 28576 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.229800 28576 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.230085 28444 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.230309 28444 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed994138812448fab5da49cea7bf73b, State: Initialized, Role: FOLLOWER
I20260806 04:34:09.230360 28348 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:09.230432 28307 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:34:09.230522 28307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:09.230603 28444 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.231657 28307 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
I20260806 04:34:09.232620 28307 leader_election.cc:288] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [CANDIDATE]: Term 1 election: Requested vote from peers eed994138812448fab5da49cea7bf73b (127.28.94.61:41319), 0b83d7d3e63b4c058a88e961807acfb4 (127.28.94.60:42919)
I20260806 04:34:09.232383 28444 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:34:09.233696 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca759490b047427bbf430d04c0d3ea05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eed994138812448fab5da49cea7bf73b"
I20260806 04:34:09.233891 28411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:09.235168 28411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca759490b047427bbf430d04c0d3ea05 in term 1.
W20260806 04:34:09.235194 28277 tablet.cc:2369] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:34:09.233240 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca759490b047427bbf430d04c0d3ea05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b83d7d3e63b4c058a88e961807acfb4"
I20260806 04:34:09.236385 28520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:34:09.237591 28274 leader_election.cc:302] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca759490b047427bbf430d04c0d3ea05, eed994138812448fab5da49cea7bf73b; no voters:
I20260806 04:34:09.238588 28520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca759490b047427bbf430d04c0d3ea05 in term 1.
I20260806 04:34:09.239238 28576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:34:09.239471 28576 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [term 1 LEADER]: Becoming Leader. State: Replica: ca759490b047427bbf430d04c0d3ea05, State: Running, Role: LEADER
I20260806 04:34:09.239779 28576 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } }
W20260806 04:34:09.235729 28597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:09.240705 28597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:09.241832 28454 master_runner.cc:263] No leader master found from master 0b83d7d3e63b4c058a88e961807acfb4
I20260806 04:34:09.241892 28454 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:34:09.242259 28576 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca759490b047427bbf430d04c0d3ea05. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.242365 28576 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:09.242465 28358 master_runner.cc:263] No leader master found from master eed994138812448fab5da49cea7bf73b
I20260806 04:34:09.242538 28358 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:34:09.242791 28643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:34:09.243709 28246 master_runner.cc:263] No leader master found from master ca759490b047427bbf430d04c0d3ea05
I20260806 04:34:09.243777 28246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:34:09.246198 28643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 04:34:09.247025 28635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:34:09.247116 28635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:34:09.247354 28598 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.247470 28598 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.252015 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.235849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.28.94.62:33101)
0806 04:34:09.235972 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.235976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.242785 (+ 6809us) server_negotiation.cc:407] Connection header received
0806 04:34:09.242992 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.242996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.243044 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.243091 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.244064 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.244781 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.245987 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.246291 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.246389 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.248253 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.248257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.248260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.248286 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.250017 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.250558 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.250562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.250564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.250658 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.250995 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.250999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.251001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.251218 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.251325 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.251718 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.251842 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:34:09.255007 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.235932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.28.94.60:42919)
0806 04:34:09.240331 (+ 4399us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.240335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.245142 (+ 4807us) server_negotiation.cc:407] Connection header received
0806 04:34:09.245319 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.245323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.245370 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.245418 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.246345 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.247091 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.248143 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.248471 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.248555 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.250493 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.250497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.250500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.250525 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.252141 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.253620 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.253624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.253626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.253717 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.254025 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.254029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.254031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.254240 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.254349 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.254729 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.254851 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4324}
I20260806 04:34:09.260619 28520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [term 1 FOLLOWER]: Refusing update from remote peer ca759490b047427bbf430d04c0d3ea05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:34:09.261003 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:09.236753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.28.94.61:41319)
0806 04:34:09.236864 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.236868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.252544 (+ 15676us) server_negotiation.cc:407] Connection header received
0806 04:34:09.252643 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.252647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.252696 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.252770 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.253541 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.254297 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.255420 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:09.255711 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:09.255788 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:09.257464 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:34:09.257468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:34:09.257471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:34:09.257499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:34:09.259073 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.259550 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.259554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.259555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.259663 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:34:09.259985 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:34:09.259989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:34:09.259991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:34:09.260199 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:34:09.260317 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:09.260655 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 04:34:09.260775 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:34:09.261008 28411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [term 1 FOLLOWER]: Refusing update from remote peer ca759490b047427bbf430d04c0d3ea05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:34:09.261348 28639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:34:09.261735 28576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:34:09.265406 28575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca759490b047427bbf430d04c0d3ea05. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.265568 28575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.265718 28598 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca759490b047427bbf430d04c0d3ea05. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.265837 28598 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.272258 28576 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.272400 28576 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:09.274099 28639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.274221 28639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05 [sys.catalog]: This master's current role is: LEADER
I20260806 04:34:09.275338 28650 mvcc.cc:205] Tried to move back new op lower bound from 7315418518563414016 to 7315418518492418048. Current Snapshot: MvccSnapshot[applied={T|T < 7315418518563414016}]
I20260806 04:34:09.275322 28575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.275429 28575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.275585 28598 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca759490b047427bbf430d04c0d3ea05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca759490b047427bbf430d04c0d3ea05" member_type: VOTER last_known_addr { host: "127.28.94.62" port: 33101 } } peers { permanent_uuid: "eed994138812448fab5da49cea7bf73b" member_type: VOTER last_known_addr { host: "127.28.94.61" port: 41319 } } peers { permanent_uuid: "0b83d7d3e63b4c058a88e961807acfb4" member_type: VOTER last_known_addr { host: "127.28.94.60" port: 42919 } } }
I20260806 04:34:09.275687 28598 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:34:09.275966 28647 mvcc.cc:205] Tried to move back new op lower bound from 7315418518563414016 to 7315418518492418048. Current Snapshot: MvccSnapshot[applied={T|T < 7315418518563414016}]
I20260806 04:34:09.276319 28643 catalog_manager.cc:1347] Generated new cluster ID: ca9d67a11b494373b3241014d71c2583
I20260806 04:34:09.276371 28643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:34:09.292711 28643 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:34:09.293970 28643 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:34:09.321203 28643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca759490b047427bbf430d04c0d3ea05: Generated new TSK 0
I20260806 04:34:09.323688 28643 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:34:09.360121 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.357593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.28.94.61:41319)
0806 04:34:09.357725 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.357729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.359022 (+ 1293us) server_negotiation.cc:407] Connection header received
0806 04:34:09.359223 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.359227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.359280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.359331 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.359948 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50678: BlockingRecv error: recv got EOF from 127.0.0.1:50678 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:34:09.361635 28571 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.358374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.28.94.60:42919)
0806 04:34:09.358490 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.358494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.360620 (+ 2126us) server_negotiation.cc:407] Connection header received
0806 04:34:09.360847 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.360850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.360920 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.360980 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.361514 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53922: BlockingRecv error: recv got EOF from 127.0.0.1:53922 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:34:09.363520 28354 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.362282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.28.94.62:33101)
0806 04:34:09.362393 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.362397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.362680 (+ 283us) server_negotiation.cc:407] Connection header received
0806 04:34:09.362839 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.362842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.362891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.362937 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.363406 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41264: BlockingRecv error: recv got EOF from 127.0.0.1:41264 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 04:34:09.367592 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.366114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.28.94.61:41319)
0806 04:34:09.366212 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.366215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.366688 (+ 473us) server_negotiation.cc:407] Connection header received
0806 04:34:09.366892 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.366895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.366980 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.367029 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.367497 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50692: BlockingRecv error: recv got EOF from 127.0.0.1:50692 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 04:34:09.372112 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.370719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.28.94.61:41319)
0806 04:34:09.370810 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.370814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.371304 (+ 490us) server_negotiation.cc:407] Connection header received
0806 04:34:09.371474 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.371477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.371524 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.371569 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.372026 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50698: BlockingRecv error: recv got EOF from 127.0.0.1:50698 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 04:34:09.377265 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.375825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.28.94.61:41319)
0806 04:34:09.375912 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.375915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.376414 (+ 499us) server_negotiation.cc:407] Connection header received
0806 04:34:09.376590 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.376593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.376636 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.376679 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.377177 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50706: BlockingRecv error: recv got EOF from 127.0.0.1:50706 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260806 04:34:09.384681 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.383161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.28.94.61:41319)
0806 04:34:09.383288 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.383292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.383802 (+ 510us) server_negotiation.cc:407] Connection header received
0806 04:34:09.384010 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.384013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.384061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.384110 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.384592 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50712: BlockingRecv error: recv got EOF from 127.0.0.1:50712 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 04:34:09.396648 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.394874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.28.94.61:41319)
0806 04:34:09.395008 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.395012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.395611 (+ 599us) server_negotiation.cc:407] Connection header received
0806 04:34:09.395833 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.395838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.395893 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.395945 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.396535 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50722: BlockingRecv error: recv got EOF from 127.0.0.1:50722 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 04:34:09.416742 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.415014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.28.94.61:41319)
0806 04:34:09.415131 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.415135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.415715 (+ 580us) server_negotiation.cc:407] Connection header received
0806 04:34:09.415969 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.415973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.416027 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.416076 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.416622 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:34:09.452764 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.451228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.28.94.61:41319)
0806 04:34:09.451372 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.451377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.451668 (+ 291us) server_negotiation.cc:407] Connection header received
0806 04:34:09.451807 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.451812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.451876 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.451963 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.452626 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50738: BlockingRecv error: recv got EOF from 127.0.0.1:50738 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 04:34:09.520435 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.519286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.28.94.61:41319)
0806 04:34:09.519447 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.519452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.519563 (+ 111us) server_negotiation.cc:407] Connection header received
0806 04:34:09.519723 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.519727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.519779 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.519849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.520329 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50748: BlockingRecv error: recv got EOF from 127.0.0.1:50748 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 04:34:09.651676 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.650509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.28.94.61:41319)
0806 04:34:09.650681 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.650686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.650783 (+ 97us) server_negotiation.cc:407] Connection header received
0806 04:34:09.650940 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.650943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.650994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.651065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.651545 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50752: BlockingRecv error: recv got EOF from 127.0.0.1:50752 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 04:34:09.910945 28453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:34:09.909837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.28.94.61:41319)
0806 04:34:09.909969 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:09.909974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:09.910062 (+ 88us) server_negotiation.cc:407] Connection header received
0806 04:34:09.910216 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:09.910219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:09.910269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:09.910338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:34:09.910824 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50764: BlockingRecv error: recv got EOF from 127.0.0.1:50764 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:34:10.242094 28597 catalog_manager.cc:1259] Loaded cluster ID: ca9d67a11b494373b3241014d71c2583
I20260806 04:34:10.242180 28597 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: loading cluster ID for follower catalog manager: success
I20260806 04:34:10.243903 28597 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: acquiring CA information for follower catalog manager: success
I20260806 04:34:10.245306 28597 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b83d7d3e63b4c058a88e961807acfb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:34:10.248178 28635 catalog_manager.cc:1259] Loaded cluster ID: ca9d67a11b494373b3241014d71c2583
I20260806 04:34:10.248235 28635 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: loading cluster ID for follower catalog manager: success
I20260806 04:34:10.249802 28635 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: acquiring CA information for follower catalog manager: success
I20260806 04:34:10.250869 28635 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eed994138812448fab5da49cea7bf73b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:34:10.429019 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:10.424989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.28.94.61:41319)
0806 04:34:10.425232 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:10.425237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:10.425330 (+ 93us) server_negotiation.cc:407] Connection header received
0806 04:34:10.425482 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:10.425485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:10.425643 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:10.425743 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:10.426000 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.426545 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.427432 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.427664 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.427751 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:10.428058 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:10.428578 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:10.428690 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:10.428753 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 04:34:10.428810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260806 04:34:10.435425 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:10.431961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.28.94.60:42919)
0806 04:34:10.432173 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:10.432177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:10.432204 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:34:10.432281 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:10.432283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:10.432410 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:10.432496 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:10.432698 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.433265 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.434086 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.434307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.434382 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:10.434620 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:10.435071 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:10.435162 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:10.435225 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 04:34:10.435273 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260806 04:34:10.442061 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:10.438265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.28.94.61:41319)
0806 04:34:10.438376 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:10.438380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:10.438783 (+ 403us) server_negotiation.cc:407] Connection header received
0806 04:34:10.438894 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:10.438897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:10.438945 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:10.439004 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:10.439358 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.439901 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441083 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.441305 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:10.441634 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:10.441715 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:10.441783 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:10.441862 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 04:34:10.441917 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:34:10.442279 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:10.438354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.28.94.60:42919)
0806 04:34:10.438453 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:10.438456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:10.439049 (+ 593us) server_negotiation.cc:407] Connection header received
0806 04:34:10.439154 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:10.439157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:10.439197 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:10.439252 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:10.439493 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.440108 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441066 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.441382 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441454 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:10.441823 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:10.441893 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:10.441934 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:10.442101 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 04:34:10.442154 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:34:10.442843 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:34:10.438149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.28.94.62:33101)
0806 04:34:10.438456 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 04:34:10.438461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:34:10.438475 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:34:10.438564 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:34:10.438566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:34:10.438672 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:34:10.438772 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:34:10.439008 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.439767 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441038 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:34:10.441382 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:34:10.441515 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:34:10.441803 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:34:10.442128 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:34:10.442604 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0806 04:34:10.442642 (+ 38us) server_negotiation.cc:299] Negotiation successful
0806 04:34:10.442701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260806 04:34:10.445595 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 28246
I20260806 04:34:10.461347 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 28358
I20260806 04:34:10.472029 29048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4FkAn1/build/debug/bin/kudu with pid 28454
2026-08-06T04:34:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3135 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6331 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (295887 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST